Qualify korvid at large-cluster scale with reproducible performance and API-load benchmarks

[hellices]

Goal

Establish a reproducible qualification program for korvid on large and noisy
clusters, then publish the measured limits and performance budgets we are
willing to support.

Current gap

korvid has broad unit/UI coverage and live-cluster write contract tests, but we
do not currently have evidence for:

• startup and first-render behavior with thousands of visible resources;
• steady-state memory and CPU under sustained watch traffic;
• event-to-screen latency during large update bursts;
• Kubernetes API request volume and accidental per-object fan-out;
• recovery from watch compaction, throttling, partial discovery, and slow APIs;
• behavior with many namespaces, CRDs, containers, and metrics samples.

Without those measurements, we cannot state a supported scale envelope or catch
performance regressions before release.

Scope

Build two complementary tiers:

1. Deterministic replay tier
   • Generate or replay realistic LIST/WATCH streams at 1k, 10k, and 50k total
     objects.
   • Include pod-heavy, CRD-heavy, namespace-heavy, and high-churn profiles.
   • Run without provisioning a large managed cluster so the core benchmark is
     repeatable by contributors and CI.
2. Live-cluster qualification tier
   • Run an opt-in workload against a dedicated non-production cluster with at
     least 1,000 simultaneously visible resources.
   • Document node count, Kubernetes version, metrics-server availability,
     network latency, RBAC profile, and workload generator settings.
   • Exercise realistic watch churn rather than only a static initial LIST.

Measure:

• process start to interactive UI and first populated table;
• input latency while LIST/WATCH and metrics updates are active;
• p50/p95 event-to-render latency;
• steady and peak RSS/CPU over a 30-minute run;
• LIST/WATCH requests, bytes, reconnects, throttles, and unexpected GET fan-out;
• time and data loss behavior after 410 Gone, disconnects, and backoff;
• filter, sort, namespace/context switch, split pane, describe, and multi-log
  behavior at scale.

Deliverables

• A versioned workload/replay format with deterministic seeds.
• A command that emits machine-readable JSON plus a human-readable summary.
• A checked-in small smoke profile and a scheduled/manual heavy profile.
• Baseline results tied to korvid SHA, Python version, OS, Kubernetes version,
  and workload profile.
• A documented support envelope and explicit performance budgets derived from
  the baseline.
• Regression checks for unbounded request fan-out and steady-state memory
  growth.

Acceptance criteria

• The replay tier covers 1k, 10k, and 50k objects and produces repeatable
  results from the same seed.
• A live non-production run with at least 1,000 visible resources is published.
• API traces demonstrate that a visible resource list does not trigger an
  unbounded per-object GET pattern.
• A 30-minute steady-state run reports memory slope, dropped updates,
  reconnects, and UI responsiveness rather than only a final pass/fail.
• Failure profiles cover API throttling, 410 Gone, partial RBAC denial,
  unavailable metrics, and slow log streams.
• The initial baseline defines numeric budgets; later regressions fail the
  relevant benchmark gate or require an explicit baseline update with rationale.
• Documentation states measured limits honestly and does not imply validation
  beyond the tested matrix.

Out of scope

• A permanent production-scale test cluster.
• Wall-clock assertions in normal unit tests.
• Optimizing before the first trace identifies the actual bottleneck.