fix(timeline): scope event-pipeline drop metrics to the active cluster - #1301
fix(timeline): scope event-pipeline drop metrics to the active cluster#1301hisco wants to merge 3 commits into
Conversation
|
Adversarial review (self + codex cross-model) of the full branch delta. Green light — contingent on landing together with #1300, after it. The contingencyStandalone, this PR does not close the diagnose-endpoint leak: Rebase note (after #1300 lands): Vetted green (checked, no action needed)
Triaged, not blocking
|
Drop records (RecentDrops) name individual resources (kind/namespace/name) and were process-global with no cluster provenance and no reset on a kubeconfig context switch. After switching clusters, the debug/diagnostics drop surfaces (/api/debug/events, /api/debug/events/diagnose, /api/diagnostics) exposed a previously-connected cluster's resource names to whoever can read the same kind on the new cluster. Stamp each DropRecord with the cluster context captured at informer wiring time (matching how timeline events are attributed), so a straggler callback firing during the asynchronous informer-shutdown window after a switch stays truthfully attributed to the cluster it came from rather than being read live. Read paths filter drops to the active cluster via timeline.DropsForCluster; GetDiagnosis scopes both its drop history and the recommendations derived from it. The per-cluster counters (which carry no resource identity) are additionally cleared on context switch via ResetMetricsForContextSwitch; process uptime is preserved. Claude-Session: https://claude.ai/code/session_01XNhMe6Zdqwj5EoBazNDnmW
92af7c1 to
c3e171b
Compare
In-cluster mode stamps and filters with ActiveClusterContext(), which returns the "in-cluster" sentinel, not an empty string. Comment said "both \"\"". Claude-Session: https://claude.ai/code/session_0164KXsqD8R5EyEUDfNkLsNb
Counters, unlike RecentDrops, have no stamp+filter guard, so a straggler informer draining after a context switch can re-increment them with a few old-cluster ticks. Accepted (kind names only, no resource identity); make the residue explicit where the reset happens. Claude-Session: https://claude.ai/code/session_0164KXsqD8R5EyEUDfNkLsNb
Problem
Event-pipeline drop records (
RecentDrops) name individual resources (kind / namespace / name). They were process-global, carried no cluster provenance, and were not reset on a kubeconfig context switch. After switching clusters, the drop surfaces —/api/debug/events,/api/debug/events/diagnose(drop_history+ its recommendations), and/api/diagnostics— exposed a previously-connected cluster's resource names to whoever can read the same kind on the new cluster.Found by cross-review of the timeline per-kind RBAC work; this is its dedicated fix.
Fix
DropRecordgains aClusterContext, captured from the informer'srecordClusterContext(the same wiring-time capture the timeline events use), passed through allRecordDropcall sites. Informer shutdown on a switch is asynchronous (up to ~5s, then abandoned), so a straggler callback can record a drop after the switch — stamping the wiring-time context keeps it attributed to the cluster it came from instead of being read live (which would mis-attribute it to the new cluster).timeline.DropsForCluster(drops, active)keeps only the active cluster's drops, applied inhandleDebugEventsand/api/diagnostics.GetDiagnosisscopes both itsDropHistoryand the recommendations derived from it.ResetMetricsForContextSwitchclears the per-cluster pipeline counters on switch (they carry no resource identity, so they only need a reset, not stamping). Process uptime is preserved — it measures the pipeline's lifetime, not a per-cluster session.Stamp+filter (not reset alone) is what closes the late-callback race — reset-at-switch-time can't stop a straggler from repopulating
RecentDropsafterward, but a straggler stamped with the old cluster is filtered out on read.Tests
internal/timeline/metrics_test.go): reset clears drops+counters while preserving uptime; a straggler drop stamped with the old cluster is filtered out of the active-cluster view.kindclusters — generate noisy-secret drops on cluster A,POST /api/contexts/<B>, assert A's resource names are gone fromRecentDropson B. Passes.go build,go vet,go test ./...green.Scope
Deliberately limited to drop-metrics cross-cluster provenance.
GetDiagnosis's timeline-store query (itsTimelineEvents) cluster-scoping is a separate change and is not touched here.https://claude.ai/code/session_01XNhMe6Zdqwj5EoBazNDnmW
Note
Medium Risk
Touches diagnostic/debug surfaces and context-switch lifecycle; mitigates information disclosure rather than changing core auth, with documented best-effort behavior on unstamped counters after async informer drain.
Overview
Fixes a cross-cluster leak where global
RecentDrops(kind/namespace/name) could show a previous kubeconfig context’s resources on debug/diagnostics APIs after a switch.DropRecordis stamped withClusterContextat informer wiring time (same as timeline events), and allRecordDropcall sites pass that context so late straggler callbacks stay attributed to the old cluster.On read,
DropsForClusterlimits drops toActiveClusterContext()in/api/debug/eventsand diagnostics;GetDiagnosisfilters drop history (and thus recommendations) by active cluster. On switch, timeline reset now also runsResetMetricsForContextSwitchto clear drops and per-cluster counters while keeping process uptime.Unit tests cover reset behavior and filtering stale stamped drops after a simulated straggler.
Reviewed by Cursor Bugbot for commit 670a750. Bugbot is set up for automated code reviews on this repo. Configure here.