Skip to content

perf(observability,do,studio): take the metrics hot-path off always-throwing DDL/COUNT, bound the sample buffer, signal truncation, batch Studio index enum - #272

Merged
prisis merged 8 commits into
alphafrom
advisor/228-observability-studio-perf
Aug 1, 2026
Merged

prisis merged 8 commits into
alphafrom
advisor/228-observability-studio-perf

Conversation

@prisis

@prisis prisis commented Aug 1, 2026 •

Copy link
Copy Markdown
Member

Five independent fixes (one commit each) across the metrics/telemetry path.

A — OBS-02: metrics DDL/COUNT off the hot path

recordFunctionMetric (once per RPC dispatch) called ensureFunctionMetricsTables unconditionally — two ALTER TABLE ADD COLUMN in try/catch that throw duplicate column name on every call after the first (two swallowed SQLite errors per invocation forever, hiding any real error), plus an unconditional SELECT COUNT(*) (no O(1) count on DO SQLite — a PK scan billed rows-read, growing with cardinality). Now: ensure* memoized per SQL handle via WeakSet (mirrors metric-history.ts); the COUNT(*) replaced with an admitPath/admitStatement known-key cache (cheap PK lookup, full count only on a genuinely new key). Same in query-metrics.ts.

Post-hibernation safety (the reviewer concern): all caches are keyed by SQL-handle object identity (WeakSet/WeakMap), so a fresh handle after hibernation starts cold and re-ensures — never skips a CREATE. Explicit regression tests in both function-metrics.test.ts and query-metrics.test.ts wrap the same storage in a brand-new handle and prove the DDL re-ensure and cap re-verification survive a simulated restart.

B — OBS-04: bound + dedupe the statement sample buffer

currentStmtSamples was an unbounded array (full raw SQL per entry); a query-in-a-loop handler accumulated one string per query then paid ~5× amplification draining it. Now a Map folded by raw text, capped at 200 distinct entries, one recordQueryMetric per distinct statement; db.stmt_samples_truncated set on the wide event when capped. recordQueryMetric/recordQueryBucket gained an additive execCount param (default 1) so a folded batch's exec_count/total_duration_ms and the latency histogram stay accurate.

C — OBS-03: truncation signal on the metric-bucket read

The all-paths bucket read truncated at LIMIT 1000 with no signal, silently shrinking the Metrics window as the app grows. Now readFunctionMetricBuckets uses a LIMIT+1 probe and returns { buckets, truncated }, threaded through getMetrics → the Studio's cross-shard SloTotals.historyTruncated (OR across shards) → a notice on the health panel.

D — OBS-05: export the barrel's types

Re-exported every barrel function's param/return types (audited all, not just the plan's examples): MetricHistoryResult, QueryInsight*, DanglingReference*, ReadRequestLogOptions/RequestLogEntry/ErrorIssue/ReadIssuesOptions, Span*, HostSpanLike, MetricEvent/MetricKind, etc. — so a second host can write a typed wrapper.

E — STUDIO-04: batch the Insights index enumeration

The advisors Insights panel issued one listTableIndexes RPC per table (41 RPCs on a 40-table app), re-run on every shard change + visibilitychange. Added a batched listTablesIndexes admin RPC (sibling of listTableIndexes, mirroring the existing describeTable→describeTables batching precedent exactly) and collapsed the fan-out to one call, keeping the per-table fallback behind the existing catch for older workers. A research pass confirmed ADMIN_FUNCTIONS is a hand-maintained string-constant object with no codegen/allowlist coupling, so the server-RPC path (not the client-side fallback) was the low-risk choice.

Verification

observability 202, do (stmt-samples + function-metrics + shard-do.admin) 134, studio (insights + health + slo-aggregate) 28 — all green. lint:types clean; api-surface gate passes (43/43 snapshots) after regenerating observability/lunora/shard-engine and studio (the last was regenerated during review — the new listTablesIndexes key needed pinning in studio.api.md).

Scope note

Section B touched query-metrics.ts (the additive execCount param) — one file outside B's literal scope but necessary to keep exec_count/duration/histogram correct under folding; minimal and backward-compatible.

Merge-order notes

Touches hot files shared with other open PRs — individually mergeable vs alpha:

Plan: plans/228-*.md.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added batched table-index inspection to improve schema loading in Studio.
    • Expanded observability exports for telemetry, query insights, request logs, metrics, and storage diagnostics.
    • SQL telemetry now tracks execution counts, totals, and distinct statements.
  • Bug Fixes

    • Improved metrics history handling with truncation detection.
    • Studio now warns when request and error trends are incomplete.
    • Optimized repeated metrics and schema lookups for better performance.

prisis and others added 6 commits August 1, 2026 03:28
recordFunctionMetric/recordQueryMetric ran on every RPC dispatch and
every instrumented statement; ensureFunctionMetricsTables re-ran two
ALTER TABLEs that throw duplicate-column after the first call, and the
distinct-key cap check paid an unconditional COUNT(*) full scan every
time. Memoize table creation per SQL handle (WeakSet, mirroring
metric-history.ts's ensuredHandles) and cache which paths/statements
are already tracked (WeakMap) so a repeat key skips both the DDL and
the count; only a genuinely new key re-verifies against durable state.
currentStmtSamples grew one array entry per raw statement execution
with no cap, so a query-in-a-loop handler accumulated unbounded memory
and paid a separate durable upsert per raw execution on flush. Fold
samples by raw statement text as they land (params never inline into
the query string, so a loop's repeats share one entry), cap distinct
statement shapes per dispatch, and record db.stmt_samples_truncated on
the wide event when the cap trips.

recordQueryMetric/recordQueryBucket gain an optional execCount
parameter (default 1, backward compatible) so a folded batch still
lands correct exec_count/total_duration_ms in the durable leaderboard
instead of undercounting; the latency histogram buckets by the
per-execution average rather than the batch sum.
readFunctionMetricBuckets's all-paths arm truncates at a fixed row
limit with no way for a caller to tell a complete window from a cut
one, so the Studio Metrics chart's range silently shrinks as an app
accumulates history, with a wrong leftmost bar and no indication why.

Detect the cut with a LIMIT+1 probe and return { buckets, truncated }
instead of a bare array. Thread historyTruncated through getMetrics
and the Studio's cross-shard SLO rollup (true if any shard was cut),
and show a notice on the health panel when it fires.
The observability barrel exported functions whose parameter and return
types were never re-exported, so a consumer typing a variable against
readFunctionMetricBuckets, readQueryInsights, readMetricHistory,
findDanglingReferences, readRequestLog/readErrorIssues, instrumentDatabase,
createMetrics, or createTracedFetch had no way to name the shape without
reaching into the package's internal modules.

Add the missing types (MetricHistoryResult, QueryInsight*, DanglingReference*,
ReadRequestLogOptions/RequestLogEntry/ErrorIssue/ReadIssuesOptions/
RequestOutcome, DatabaseTelemetryDeps, MetricEvent/MetricKind,
SpanEvent/SpanKind/SpanOptions/SpanLink/SpanEventPoint, HostSpanLike/
HostTracingResolver, TracedFetchDeps, FoldedTraces) to the barrel.
insights-panel.tsx issued one listTableIndexes admin RPC PER table on
every shard change and visibilitychange refresh — 41 RPCs on a 40-table
app for a single panel load.

Add listTablesIndexes, a batched sibling of listTableIndexes mirroring
the existing describeTables/describeTable pair (tableIndexes is a
cheap, synchronous, schema-sourced lookup, so this is purely a
round-trip-count win). The panel now issues one call for every table
and falls back to the old per-table fan-out when the batched RPC is
unavailable (older worker) or fails outright.
The batched index-enumeration admin RPC added a new key to the exported
ADMIN_FUNCTIONS const; regenerate the studio snapshot so the api-surface
gate matches the built output.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@netlify

netlify Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit 347838f
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a6dc9d3d7e21800089e33b5
😎 Deploy Preview https://deploy-preview-272--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@prisis, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 9 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: ebba836e-ced2-445f-84e4-2f68694b500e

📥 Commits

Reviewing files that changed from the base of the PR and between 3ebb45f and 347838f.

⛔ Files ignored due to path filters (3)
  • packages/observability/__tests__/_helpers/fresh-handle.ts is excluded by !**/__tests__/** and included by packages/**
  • packages/observability/__tests__/function-metrics.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/observability/__tests__/query-metrics.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (1)
  • packages/studio/src/features/advisors/insights-panel.tsx

Walkthrough

The change aggregates SQL telemetry by statement, persists execution totals, reports truncated metric history, caches repeated metric checks, and adds batched table-index introspection with Studio fallback handling.

Changes

Telemetry aggregation and history

Layer / File(s) Summary
Metric storage and admission
packages/observability/src/context-telemetry.ts, packages/observability/src/function-metrics.ts, packages/observability/src/query-metrics.ts, packages/observability/src/index.ts
Metric initialization and admission checks are cached per SQL handle. Query metrics accept folded execution counts. Function metric reads return bounded buckets with a truncated flag.
Shard SQL aggregation and persistence
packages/do/src/shard-do.ts
ShardDO folds repeated statements, enforces a 200-statement cap, records sampling truncation, and persists aggregate counts, durations, and row totals.
Truncated history reporting
packages/studio/src/features/reports/slo-aggregate.ts, packages/studio/src/features/reports/health-panel.tsx, packages/studio/src/lib/admin.ts, packages/studio/src/locales/en.ts
Studio propagates truncation across shard metrics and displays a warning for incomplete request/error history.

Batched table-index introspection

Layer / File(s) Summary
Batched index RPC contract and handler
packages/shard-engine/src/introspect.ts, packages/shard-engine/src/index.ts, packages/do/src/shard-do.ts
The admin API adds listTablesIndexes and returns indexes grouped by table. Shard introspection uses shared batching for table metadata lookups.
Studio index enumeration and fallback
packages/studio/src/lib/admin.ts, packages/studio/src/features/advisors/insights-panel.tsx
Studio performs one index request per shard and falls back to per-table requests when the batched RPC fails.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SQLExecution
  participant ShardDO
  participant QueryMetrics
  participant DurableHistory
  participant Studio
  SQLExecution->>ShardDO: Record statement execution
  ShardDO->>ShardDO: Aggregate executions and enforce cap
  ShardDO->>QueryMetrics: Persist statement totals and execCount
  QueryMetrics->>DurableHistory: Update lifetime and bucket metrics
  Studio->>ShardDO: Request metric history
  ShardDO-->>Studio: Return buckets and historyTruncated
  Studio->>Studio: Aggregate shard truncation flags
  Studio-->>Studio: Display partial-history warning
Loading
sequenceDiagram
  participant Studio
  participant ShardDO
  participant Introspection
  Studio->>ShardDO: Request indexes for all tables
  ShardDO->>Introspection: Perform batched index lookup
  Introspection-->>ShardDO: Return indexesByTable
  ShardDO-->>Studio: Return grouped indexes
  Studio->>Studio: Use per-table fallback if batch request fails
Loading

Possibly related PRs

  • anolilab/lunora#158: Extends durable metric-history and observability work in the same telemetry paths.
  • anolilab/lunora#184: Relates to memoized metric-history initialization and per-handle checks.
  • anolilab/lunora#200: Relates to Studio handling of metric-history results and truncated history.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the related performance, truncation, and Studio batching changes.
Description check ✅ Passed The description provides detailed scope, rationale, verification, and merge notes, although it does not follow all template headings.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch advisor/228-observability-studio-perf

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@github-actions

github-actions Bot commented Aug 1, 2026 •

Copy link
Copy Markdown
Contributor

React Doctor found 3 new issues in 1 file · 3 warnings · score 92 / 100 (Great) · 0 fixed · vs alpha

3 warnings

src/features/advisors/insights-panel.tsx

  • ⚠️ L139 Large component is hard to read and change no-giant-component
  • ⚠️ L386 Redundant manual memoization react-compiler-no-manual-memoization
  • ⚠️ L417 Redundant manual memoization react-compiler-no-manual-memoization

Reviewed by React Doctor for commit 347838f. See inline comments for fixes.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
packages/observability/src/query-metrics.ts (1)

128-162: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the repeated "ensured table" and "admit under cap" caching idioms into a shared helper.

This file implements the same WeakSet-gated "ensure table once" idiom twice (ensuredMetricsHandles/ensureQueryMetricsTable and ensuredBucketsHandles/ensureQueryBucketsTable), and the same WeakMap-gated "admit under cap" idiom (knownStatements/admitStatement) that function-metrics.ts also implements independently (knownPaths/admitPath). The logic is structurally identical in all cases: a WeakSet<SqlExec> (or WeakMap<SqlExec, Set<string>>) memoizes a one-time-per-handle SQL check, with only the table/column names and cap constant differing.

Extract a small generic helper (for example ensureTableOnce(sql, gate, createFn) and admitWithCap(sql, knownSet, existsQuery, countQuery, cap)) into a shared module. This keeps the cap-check and memoization semantics correct in one place instead of four separate copies that must be kept in sync by hand.

As per path instructions, "Review TypeScript code following the project's coding guidelines: ... Follow DRY principles" for packages/**/*.ts.

Also applies to: 211-245, 447-501

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/observability/src/query-metrics.ts` around lines 128 - 162, Extract
the duplicated per-handle table initialization and capped-admission logic into
shared generic helpers, preserving the existing WeakSet/WeakMap memoization and
SQL-check semantics. Update ensureQueryMetricsTable, ensureQueryBucketsTable,
and the corresponding knownStatements/admitStatement and knownPaths/admitPath
implementations to use those helpers, parameterizing only SQL identifiers,
queries, and cap values.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/studio/src/features/advisors/insights-panel.tsx`:
- Around line 328-333: Update the enumeration flow around enumerateShard so it
reads the latest loadShardTraffic-backed loader through a ref or equivalent
stable latest-value mechanism, rather than retaining the initial
loadShardTrafficFeed closure when client is unchanged. Preserve the existing
fixed [client] dependency key and ensure later enumerations use the updated
injected loader.

---

Nitpick comments:
In `@packages/observability/src/query-metrics.ts`:
- Around line 128-162: Extract the duplicated per-handle table initialization
and capped-admission logic into shared generic helpers, preserving the existing
WeakSet/WeakMap memoization and SQL-check semantics. Update
ensureQueryMetricsTable, ensureQueryBucketsTable, and the corresponding
knownStatements/admitStatement and knownPaths/admitPath implementations to use
those helpers, parameterizing only SQL identifiers, queries, and cap values.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3e4fd955-f993-4303-9b7d-6876f584e5c9

📥 Commits

Reviewing files that changed from the base of the PR and between 36421ad and 3ebb45f.

⛔ Files ignored due to path filters (12)
  • api-snapshots/lunora.api.md is excluded by none and included by none
  • api-snapshots/observability.api.md is excluded by none and included by none
  • api-snapshots/shard-engine.api.md is excluded by none and included by none
  • api-snapshots/studio.api.md is excluded by none and included by none
  • packages/do/__tests__/function-metrics.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/do/__tests__/shard-do.admin.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/do/__tests__/stmt-samples.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/observability/__tests__/function-metrics.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/observability/__tests__/query-metrics.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/studio/__tests__/features/advisors/insights-panel.test.tsx is excluded by !**/__tests__/** and included by packages/**
  • packages/studio/__tests__/features/reports/health-panel.test.tsx is excluded by !**/__tests__/** and included by packages/**
  • packages/studio/__tests__/features/reports/slo-aggregate.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (12)
  • packages/do/src/shard-do.ts
  • packages/observability/src/context-telemetry.ts
  • packages/observability/src/function-metrics.ts
  • packages/observability/src/index.ts
  • packages/observability/src/query-metrics.ts
  • packages/shard-engine/src/index.ts
  • packages/shard-engine/src/introspect.ts
  • packages/studio/src/features/advisors/insights-panel.tsx
  • packages/studio/src/features/reports/health-panel.tsx
  • packages/studio/src/features/reports/slo-aggregate.ts
  • packages/studio/src/lib/admin.ts
  • packages/studio/src/locales/en.ts

@codspeed

codspeed Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 251 untouched benchmarks
⏩ 10 skipped benchmarks1


Comparing advisor/228-observability-studio-perf (347838f) with alpha (a540991)2

Open in CodSpeed

Footnotes

  1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on alpha (d9791ea) during the generation of this report, so a540991 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

freshHandleOver (the fake SqlExec-over-shared-storage simulating a
post-hibernation fresh isolate) was copy-pasted verbatim into both
function-metrics.test.ts and query-metrics.test.ts. Hoist it into
__tests__/_helpers/fresh-handle.ts, next to the existing node-sqlite
harness, and import it from both suites.
Turn enumerateShard into an effect event instead of a useCallback pinned
to [client]. It closes over loadShardTraffic/fanShardTraffic through
render-fresh helpers, so a memoized callback kept whichever override
was in scope when it was (re)created — an injected loadShardTraffic
change with client unchanged silently kept fanning traffic through the
stale override. An effect event always resolves the latest committed
closures and stays out of the effect's dependency array, which also
removes the manual memoization React Compiler already subsumes.
@prisis
prisis merged commit 0de2fb4 into alpha Aug 1, 2026
44 of 46 checks passed
@prisis
prisis deleted the advisor/228-observability-studio-perf branch August 1, 2026 10:56
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — 347838f3 Deployed Aug 1, 2026 by prisis via Benchmarks #1326
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant