Skip to content

fix(observability): correct the metric-cap rationale to the post-guard threat model - #304

Closed
prisis wants to merge 7 commits into
alphafrom
advisor/276-observability-hardening
Closed

prisis wants to merge 7 commits into
alphafrom
advisor/276-observability-hardening

Conversation

@prisis

@prisis prisis commented Aug 2, 2026

Copy link
Copy Markdown
Member

Branched before #288/#289 landed, so it needs a rebase onto current alpha before merge.

Commits

  • fix(observability): correct the metric-cap rationale to the post-guard threat model
  • fix(observability): evict the least-recently-updated series/path/statement at the durable metric caps
  • fix(observability): stringify non-JSON-able failures and redact database-span error messages
  • fix(observability): redact span error messages and gate stacktraces like the request log
  • fix(observability): drop-on-full instead of evict at the durable metric caps
  • fix(observability): stop exporting describeFailure solely for test access
  • chore(observability): update API snapshot for this branch's surface changes

Files this branch still changes relative to alpha

api-snapshots/observability.api.md
packages/do/__tests__/function-metrics.test.ts
packages/do/src/shard-do.ts
packages/observability/__tests__/context-telemetry.test.ts
packages/observability/__tests__/database-telemetry.test.ts
packages/observability/__tests__/function-metrics.test.ts
packages/observability/__tests__/metric-history.test.ts
packages/observability/__tests__/query-metrics.test.ts
packages/observability/src/context-telemetry.ts
packages/observability/src/database-telemetry.ts
packages/observability/src/function-metrics.ts
packages/observability/src/metric-history.ts
packages/observability/src/query-metrics.ts

prisis added 7 commits August 1, 2026 18:02
…d threat model

function-metrics.ts's FUNCTION_METRICS_MAX_PATHS docstring claimed an
unregistered/FUNCTION_NOT_FOUND dispatch "still records a row" — but
shard-do.ts's dispatch handler explicitly guards against exactly that,
so the stated attacker-reachability rationale for the cap no longer
matches the code. metric-history.ts's module docstring inherited the
same wrong framing via its "mirrors function-metrics.ts" cross-reference.

Rewrite both to name the real bound (the app's own registered-function
set plus deploy churn) and to describe the LRU-at-capacity eviction
policy that the next commit introduces.
…ement at the durable metric caps

The three durable metric caps (metric-history, function-metrics,
query-metrics) were one-way ratchets: at the cap, a brand-new
series/path/statement was silently dropped forever while a dead one kept
its budget slot indefinitely. A shard that hit its cap went permanently
blind to new identities — including the app's own new metrics after a
deploy, and including the incident an operator is trying to chart. The
in-memory twin (MetricBuffer) already evicts the least-recently-updated
entry instead; the durable side had the opposite failure mode for the
same cardinality problem.

Mirror MetricBuffer's LRU-at-capacity policy durably in all three
modules: at the cap, delete the identity with the oldest recency key
(last_ts / last_called_at / the newly added last_seen_at) and its
satellite rows, then admit the new one. Already-tracked identities keep
accumulating past the cap unchanged.

query-metrics' lifetime table had no recency column, so add a nullable
last_seen_at via a guarded ALTER TABLE (mirroring function-metrics'
existing scans/conflicts backfill), maintained by the upsert and ordered
NULLS FIRST so legacy rows evict before any row with a real timestamp.

Also add a capped boolean to readMetricHistory's result, mirroring
readQueryInsights/readFunctionMetricBuckets's existing truncation
signals, so the Studio can say "showing N of a capped set" instead of
implying totality.
…ase-span error messages

Two related database-telemetry defects:

- describeFailure's non-Error/non-string fallback was bare
  JSON.stringify(failure), which is typed => string but returns
  undefined for a function/symbol/undefined value — landing the literal
  undefined in a SpanEvent["error"].message field declared string, then
  encoded as an OTLP exception.message attribute. Fall back to String()
  the same way request-log.ts's renderLogMessage already does.

- A failed ctx.db call's error message (a constraint breach quotes the
  conflicting row) was recorded on the CLIENT span verbatim, while the
  request log and function-metrics sinks already redact their error
  messages by default. Thread a captureRaw flag (default false) into
  DatabaseTelemetryDeps and redact at the recording boundary with the
  same redactArgs the request log uses, so production stays redacted
  and only the dev escape hatch (captureRaw: true) sees raw messages.
…ike the request log

Raw error messages and stack traces shipped verbatim to third-party
collectors through the span pipeline (ctx.trace's caught-error message,
the per-dispatch root span's failure message, and recordException's
exception event), while the request log and function-metrics sinks
already redact their error messages by default. The same validation
error masked in __lunora_reqlog__ and the Studio panel shipped in the
clear to whatever otlpSink/webhookSink an operator configured — the
sink with the longest retention.

Thread a captureRaw flag (default false) into TracerDeps,
createSpanCollector, and dispatchRootSpan's input, and wrap every
message assignment with the same redactArgs the request log uses.
exception.stacktrace is now captureRaw-gated too — a stack is
file-paths-and-internals by definition, the exact class isInternalCode
redaction exists for.

shard-do.ts threads isDevEnvironment(this.env) at all four call sites
(makeTracer, instrumentDb, makeDispatchSpan's collector, and
recordDispatchRootSpan's dispatchRootSpan call), matching the existing
request-log exemplar: raw in dev, redacted in production. Confirmed no
consumer needs raw production stacktraces or span error messages for
correctness — fingerprintError groups off the raw thrown error captured
at the request-log recording boundary, not off recorded spans.
…ic caps

Thermos review of the eviction policy (058d4a8): it inverted what the
caps were protecting. The old drop-on-full behaviour protected
incumbents — a flood of one-off statement shapes/paths/series was
refused, and the app's real leaderboard survived untouched. LRU eviction
inverted that: a flood now evicts the app's real tracked identities,
which for an observability store is the worse failure mode (it silently
empties Studio's Query Insights / metric charts / function stats rather
than merely failing to add to them) and hands an operator a
monitoring-evasion primitive that didn't exist before. It also put an
unbounded-cost sort (ORDER BY on an unindexed recency column, plus two
extra DELETEs) on every over-cap write, replacing a cheap early return.

The plan's actual complaint was narrower: the drop was silent, with no
read-side signal an operator could see. Revert the eviction logic in
all three modules back to refuse-on-full, and keep only the missing
signal:

- metric-history: `readMetricHistory`'s new `capped` flag stays (this
  module had no signal at all before).
- function-metrics: add the same `capped` flag to
  `readFunctionMetricsTotals`, which had a `truncated` read-limit signal
  but no write-cap one.
- query-metrics: `readQueryInsights`'s `capped` already existed and is
  unchanged. Remove the `last_seen_at` column/ALTER TABLE/extra upsert
  write added to support eviction — it has no remaining purpose and was
  the source of the unindexed-sort cost.

All three `admit*` helpers now consistently return `boolean` and every
caller honours it (metric-history's admission check previously returned
`void` and was ignored, silently turning METRIC_HISTORY_MAX_SERIES into
a no-op cap whenever eviction "succeeded").

Also fixes `readMetricHistory`'s `capped` docblock: it's a write-side
signal computed over the whole table, independent of `sinceMs`/the read
limit — not a read-truncation flag, despite reading similarly to
`truncated` elsewhere.
…cess

describeFailure had zero production consumers outside the module — it
was exported only so the previous commit's unit tests could import it
directly. Un-export it and reach it through instrumentDatabase (its one
real caller) instead, matching how the rest of this module's helpers
are tested.

The bare-undefined-rejection case is no longer exercised directly:
buildDatabaseSpan treats `failure === undefined` as "no failure" (the
same sentinel createTracer's own `error` variable uses), so a literal
`undefined` rejection never reaches describeFailure through the public
path. That's a separate, pre-existing limitation of the sentinel
pattern, not something this change touches — recorded in the test's
comment rather than worked around.
…hanges

pnpm run api:update after rebuilding — api:check is invisible to
ESLint/Prettier/lint:types and would have failed CI on the
captureRaw/capped additions across TracerDeps, DatabaseTelemetryDeps,
MetricHistoryResult, readMetricHistory, readFunctionMetricsTotals,
createSpanCollector, and dispatchRootSpan.
@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@prisis, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 59 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: fb751f5f-7892-403c-8f73-4ada8308c984

📥 Commits

Reviewing files that changed from the base of the PR and between 6a4dd72 and 3a22233.

⛔ Files ignored due to path filters (7)
  • api-snapshots/observability.api.md is excluded by none and included by none
  • packages/do/__tests__/function-metrics.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/observability/__tests__/context-telemetry.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/observability/__tests__/database-telemetry.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/observability/__tests__/function-metrics.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/observability/__tests__/metric-history.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/observability/__tests__/query-metrics.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (6)
  • packages/do/src/shard-do.ts
  • packages/observability/src/context-telemetry.ts
  • packages/observability/src/database-telemetry.ts
  • packages/observability/src/function-metrics.ts
  • packages/observability/src/metric-history.ts
  • packages/observability/src/query-metrics.ts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@netlify

netlify Bot commented Aug 2, 2026 •

Copy link
Copy Markdown

❌ Deploy Preview for lunorash failed.

Name Link
🔨 Latest commit 3a22233
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a6f0b5ecbe5ac0007640bce

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@codspeed

codspeed Bot commented Aug 2, 2026

Copy link
Copy Markdown

This PR has no benchmarks

⚠️ No benchmarks were detected in both the base of the PR and the PR.
Please ensure that your benchmarks are correctly instrumented with CodSpeed.

Check out the benchmarks creation guide


Comparing advisor/276-observability-hardening (3a22233) with alpha (6a4dd72)

Open in CodSpeed

@prisis

prisis commented Aug 2, 2026

Copy link
Copy Markdown
Member Author

Closing: this PR was opened in error by an automated run, outside the scope of its task.

No work is lost — the branch is retained on origin and this PR can be reopened at any time.

Context: this branch predates the merges of #288 and #289, so much of its content is already on alpha, and it would need a rebase before being reviewable. Reopening deliberately, after a rebase, is preferable to leaving it open in this state.

@prisis prisis closed this Aug 2, 2026
@prisis
prisis deleted the advisor/276-observability-hardening branch August 2, 2026 09:46
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — 3a222335 Deployed Aug 2, 2026 by prisis via Benchmarks #1367
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant