Repository navigation
perf(observability,do,studio): take the metrics hot-path off always-throwing DDL/COUNT, bound the sample buffer, signal truncation, batch Studio index enum - #272
Conversation
recordFunctionMetric/recordQueryMetric ran on every RPC dispatch and every instrumented statement; ensureFunctionMetricsTables re-ran two ALTER TABLEs that throw duplicate-column after the first call, and the distinct-key cap check paid an unconditional COUNT(*) full scan every time. Memoize table creation per SQL handle (WeakSet, mirroring metric-history.ts's ensuredHandles) and cache which paths/statements are already tracked (WeakMap) so a repeat key skips both the DDL and the count; only a genuinely new key re-verifies against durable state.
currentStmtSamples grew one array entry per raw statement execution with no cap, so a query-in-a-loop handler accumulated unbounded memory and paid a separate durable upsert per raw execution on flush. Fold samples by raw statement text as they land (params never inline into the query string, so a loop's repeats share one entry), cap distinct statement shapes per dispatch, and record db.stmt_samples_truncated on the wide event when the cap trips. recordQueryMetric/recordQueryBucket gain an optional execCount parameter (default 1, backward compatible) so a folded batch still lands correct exec_count/total_duration_ms in the durable leaderboard instead of undercounting; the latency histogram buckets by the per-execution average rather than the batch sum.
readFunctionMetricBuckets's all-paths arm truncates at a fixed row
limit with no way for a caller to tell a complete window from a cut
one, so the Studio Metrics chart's range silently shrinks as an app
accumulates history, with a wrong leftmost bar and no indication why.
Detect the cut with a LIMIT+1 probe and return { buckets, truncated }
instead of a bare array. Thread historyTruncated through getMetrics
and the Studio's cross-shard SLO rollup (true if any shard was cut),
and show a notice on the health panel when it fires.
The observability barrel exported functions whose parameter and return types were never re-exported, so a consumer typing a variable against readFunctionMetricBuckets, readQueryInsights, readMetricHistory, findDanglingReferences, readRequestLog/readErrorIssues, instrumentDatabase, createMetrics, or createTracedFetch had no way to name the shape without reaching into the package's internal modules. Add the missing types (MetricHistoryResult, QueryInsight*, DanglingReference*, ReadRequestLogOptions/RequestLogEntry/ErrorIssue/ReadIssuesOptions/ RequestOutcome, DatabaseTelemetryDeps, MetricEvent/MetricKind, SpanEvent/SpanKind/SpanOptions/SpanLink/SpanEventPoint, HostSpanLike/ HostTracingResolver, TracedFetchDeps, FoldedTraces) to the barrel.
insights-panel.tsx issued one listTableIndexes admin RPC PER table on every shard change and visibilitychange refresh — 41 RPCs on a 40-table app for a single panel load. Add listTablesIndexes, a batched sibling of listTableIndexes mirroring the existing describeTables/describeTable pair (tableIndexes is a cheap, synchronous, schema-sourced lookup, so this is purely a round-trip-count win). The panel now issues one call for every table and falls back to the old per-table fan-out when the batched RPC is unavailable (older worker) or fails outright.
The batched index-enumeration admin RPC added a new key to the exported ADMIN_FUNCTIONS const; regenerate the studio snapshot so the api-surface gate matches the built output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Warning Review limit reached
Next review available in: 9 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (3)
📒 Files selected for processing (1)
WalkthroughThe change aggregates SQL telemetry by statement, persists execution totals, reports truncated metric history, caches repeated metric checks, and adds batched table-index introspection with Studio fallback handling. ChangesTelemetry aggregation and history
Batched table-index introspection
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant SQLExecution
participant ShardDO
participant QueryMetrics
participant DurableHistory
participant Studio
SQLExecution->>ShardDO: Record statement execution
ShardDO->>ShardDO: Aggregate executions and enforce cap
ShardDO->>QueryMetrics: Persist statement totals and execCount
QueryMetrics->>DurableHistory: Update lifetime and bucket metrics
Studio->>ShardDO: Request metric history
ShardDO-->>Studio: Return buckets and historyTruncated
Studio->>Studio: Aggregate shard truncation flags
Studio-->>Studio: Display partial-history warning
sequenceDiagram
participant Studio
participant ShardDO
participant Introspection
Studio->>ShardDO: Request indexes for all tables
ShardDO->>Introspection: Perform batched index lookup
Introspection-->>ShardDO: Return indexesByTable
ShardDO-->>Studio: Return grouped indexes
Studio->>Studio: Use per-table fallback if batch request fails
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for following the naming conventions! 🙏 |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
|
React Doctor found 3 new issues in 1 file · 3 warnings · score 92 / 100 (Great) · 0 fixed · vs 3 warnings
Reviewed by React Doctor for commit |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/observability/src/query-metrics.ts (1)
128-162: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the repeated "ensured table" and "admit under cap" caching idioms into a shared helper.
This file implements the same WeakSet-gated "ensure table once" idiom twice (
ensuredMetricsHandles/ensureQueryMetricsTableandensuredBucketsHandles/ensureQueryBucketsTable), and the same WeakMap-gated "admit under cap" idiom (knownStatements/admitStatement) thatfunction-metrics.tsalso implements independently (knownPaths/admitPath). The logic is structurally identical in all cases: aWeakSet<SqlExec>(orWeakMap<SqlExec, Set<string>>) memoizes a one-time-per-handle SQL check, with only the table/column names and cap constant differing.Extract a small generic helper (for example
ensureTableOnce(sql, gate, createFn)andadmitWithCap(sql, knownSet, existsQuery, countQuery, cap)) into a shared module. This keeps the cap-check and memoization semantics correct in one place instead of four separate copies that must be kept in sync by hand.As per path instructions, "Review TypeScript code following the project's coding guidelines: ... Follow DRY principles" for
packages/**/*.ts.Also applies to: 211-245, 447-501
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/observability/src/query-metrics.ts` around lines 128 - 162, Extract the duplicated per-handle table initialization and capped-admission logic into shared generic helpers, preserving the existing WeakSet/WeakMap memoization and SQL-check semantics. Update ensureQueryMetricsTable, ensureQueryBucketsTable, and the corresponding knownStatements/admitStatement and knownPaths/admitPath implementations to use those helpers, parameterizing only SQL identifiers, queries, and cap values.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/studio/src/features/advisors/insights-panel.tsx`:
- Around line 328-333: Update the enumeration flow around enumerateShard so it
reads the latest loadShardTraffic-backed loader through a ref or equivalent
stable latest-value mechanism, rather than retaining the initial
loadShardTrafficFeed closure when client is unchanged. Preserve the existing
fixed [client] dependency key and ensure later enumerations use the updated
injected loader.
---
Nitpick comments:
In `@packages/observability/src/query-metrics.ts`:
- Around line 128-162: Extract the duplicated per-handle table initialization
and capped-admission logic into shared generic helpers, preserving the existing
WeakSet/WeakMap memoization and SQL-check semantics. Update
ensureQueryMetricsTable, ensureQueryBucketsTable, and the corresponding
knownStatements/admitStatement and knownPaths/admitPath implementations to use
those helpers, parameterizing only SQL identifiers, queries, and cap values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 3e4fd955-f993-4303-9b7d-6876f584e5c9
⛔ Files ignored due to path filters (12)
api-snapshots/lunora.api.mdis excluded by none and included by noneapi-snapshots/observability.api.mdis excluded by none and included by noneapi-snapshots/shard-engine.api.mdis excluded by none and included by noneapi-snapshots/studio.api.mdis excluded by none and included by nonepackages/do/__tests__/function-metrics.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/do/__tests__/shard-do.admin.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/do/__tests__/stmt-samples.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/observability/__tests__/function-metrics.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/observability/__tests__/query-metrics.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/studio/__tests__/features/advisors/insights-panel.test.tsxis excluded by!**/__tests__/**and included bypackages/**packages/studio/__tests__/features/reports/health-panel.test.tsxis excluded by!**/__tests__/**and included bypackages/**packages/studio/__tests__/features/reports/slo-aggregate.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (12)
packages/do/src/shard-do.tspackages/observability/src/context-telemetry.tspackages/observability/src/function-metrics.tspackages/observability/src/index.tspackages/observability/src/query-metrics.tspackages/shard-engine/src/index.tspackages/shard-engine/src/introspect.tspackages/studio/src/features/advisors/insights-panel.tsxpackages/studio/src/features/reports/health-panel.tsxpackages/studio/src/features/reports/slo-aggregate.tspackages/studio/src/lib/admin.tspackages/studio/src/locales/en.ts
Merging this PR will not alter performance
Comparing Footnotes
|
freshHandleOver (the fake SqlExec-over-shared-storage simulating a post-hibernation fresh isolate) was copy-pasted verbatim into both function-metrics.test.ts and query-metrics.test.ts. Hoist it into __tests__/_helpers/fresh-handle.ts, next to the existing node-sqlite harness, and import it from both suites.
Turn enumerateShard into an effect event instead of a useCallback pinned to [client]. It closes over loadShardTraffic/fanShardTraffic through render-fresh helpers, so a memoized callback kept whichever override was in scope when it was (re)created — an injected loadShardTraffic change with client unchanged silently kept fanning traffic through the stale override. An effect event always resolves the latest committed closures and stays out of the effect's dependency array, which also removes the manual memoization React Compiler already subsumes.
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Five independent fixes (one commit each) across the metrics/telemetry path.
A — OBS-02: metrics DDL/COUNT off the hot path
recordFunctionMetric(once per RPC dispatch) calledensureFunctionMetricsTablesunconditionally — twoALTER TABLE ADD COLUMNin try/catch that throwduplicate column nameon every call after the first (two swallowed SQLite errors per invocation forever, hiding any real error), plus an unconditionalSELECT COUNT(*)(no O(1) count on DO SQLite — a PK scan billed rows-read, growing with cardinality). Now:ensure*memoized per SQL handle viaWeakSet(mirrorsmetric-history.ts); theCOUNT(*)replaced with anadmitPath/admitStatementknown-key cache (cheap PK lookup, full count only on a genuinely new key). Same inquery-metrics.ts.Post-hibernation safety (the reviewer concern): all caches are keyed by SQL-handle object identity (
WeakSet/WeakMap), so a fresh handle after hibernation starts cold and re-ensures — never skips aCREATE. Explicit regression tests in bothfunction-metrics.test.tsandquery-metrics.test.tswrap the same storage in a brand-new handle and prove the DDL re-ensure and cap re-verification survive a simulated restart.B — OBS-04: bound + dedupe the statement sample buffer
currentStmtSampleswas an unbounded array (full raw SQL per entry); a query-in-a-loop handler accumulated one string per query then paid ~5× amplification draining it. Now aMapfolded by raw text, capped at 200 distinct entries, onerecordQueryMetricper distinct statement;db.stmt_samples_truncatedset on the wide event when capped.recordQueryMetric/recordQueryBucketgained an additiveexecCountparam (default 1) so a folded batch'sexec_count/total_duration_msand the latency histogram stay accurate.C — OBS-03: truncation signal on the metric-bucket read
The all-paths bucket read truncated at
LIMIT 1000with no signal, silently shrinking the Metrics window as the app grows. NowreadFunctionMetricBucketsuses aLIMIT+1probe and returns{ buckets, truncated }, threaded throughgetMetrics→ the Studio's cross-shardSloTotals.historyTruncated(OR across shards) → a notice on the health panel.D — OBS-05: export the barrel's types
Re-exported every barrel function's param/return types (audited all, not just the plan's examples):
MetricHistoryResult,QueryInsight*,DanglingReference*,ReadRequestLogOptions/RequestLogEntry/ErrorIssue/ReadIssuesOptions,Span*,HostSpanLike,MetricEvent/MetricKind, etc. — so a second host can write a typed wrapper.E — STUDIO-04: batch the Insights index enumeration
The advisors Insights panel issued one
listTableIndexesRPC per table (41 RPCs on a 40-table app), re-run on every shard change +visibilitychange. Added a batchedlistTablesIndexesadmin RPC (sibling oflistTableIndexes, mirroring the existingdescribeTable→describeTablesbatching precedent exactly) and collapsed the fan-out to one call, keeping the per-table fallback behind the existing catch for older workers. A research pass confirmedADMIN_FUNCTIONSis a hand-maintained string-constant object with no codegen/allowlist coupling, so the server-RPC path (not the client-side fallback) was the low-risk choice.Verification
observability 202, do (stmt-samples + function-metrics + shard-do.admin) 134, studio (insights + health + slo-aggregate) 28 — all green.
lint:typesclean; api-surface gate passes (43/43 snapshots) after regeneratingobservability/lunora/shard-engineandstudio(the last was regenerated during review — the newlistTablesIndexeskey needed pinning instudio.api.md).Scope note
Section B touched
query-metrics.ts(the additiveexecCountparam) — one file outside B's literal scope but necessary to keepexec_count/duration/histogram correct under folding; minimal and backward-compatible.Merge-order notes
Touches hot files shared with other open PRs — individually mergeable vs
alpha:packages/do/src/shard-do.ts(also in the shard-engine/DO PRs fix(shard-engine): string gt() under-invalidation + close transaction-meter gaps #243, fix(runtime): base64url-encode the forwarded identity header (non-Latin-1 claims break every RPC) #262) — the edits here are the metrics/admin paths; resolve by keeping both sides.packages/shard-engine/src/index.ts(also feat(shard-engine): surface FTS score and geo distance via collectWithScores #267) — both append exports; keep both.Plan:
plans/228-*.md.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes