Skip to content

feat(observability)!: sampling coherence, health, log-archive, DO telemetry + OTLP (Wave 15: 179,180,181,186,190) - #198

Merged
prisis merged 17 commits into
alphafrom
advisor/wave15-observability
Jul 26, 2026
Merged

prisis merged 17 commits into
alphafrom
advisor/wave15-observability

Conversation

@prisis

@prisis prisis commented Jul 26, 2026

Copy link
Copy Markdown
Member

Wave 15 — observability, sampling & OTLP correctness (plans 179, 180, 181, 186, 190)

Five reviewed plans in the telemetry/runtime/DO surface, merged into one branch (all touch create-worker.ts / shard-do.ts / observability.ts — conflicts resolved and the combined suite re-run).

  • 179 (OBS-01) — settle the trace-sampling verdict once: the RPC-event export gate now consumes the dispatch's already-settled verdict instead of re-deriving it, so the propagated traceparent and the exported SERVER span can never disagree (re-fixes the orphan-span class from fix(do): propagate the inbound sampled bit instead of hardcoding it #191). +agreement tests.
  • 180 (OBS-02/03) — /_lunora/health hardening: the public posture no longer echoes operator binding names (recon leak vs the module's own invariant), and cacheTtlMs is now real (was a documented no-op that left the unauthenticated endpoint running live DO/D1 probes per request).
  • 181 (OBS-04) — lossless durable log-archive pagination: the keyset cursor dropped ≥1 record at every page boundary. Now inclusive ts <= resume + last-returned-row cursor + seen-hash dedup + degenerate-page grow. Completeness test proves union-of-pages == full result.
  • 186 (OBS-07/08/09) — DO telemetry hot path: metricHistory is now an opt-out (default off — it was an unconditional durable SQLite write per ctx.metrics.*); sampled-out held spans live on the per-trace entry (no whole-ring copy per dispatch; eviction race closed); a span ending after its sampled-out dispatch tore down no longer exports unconditionally (verdict snapshot).
  • 190 (OBS-10/11/12, HLTH-03) — OTLP export: reserved attribute keys converged across DO + worker via LUNORA_ATTR (wire-format change — see fix(otlp)!); gzip decided by byte length (not UTF-16 units); resource grouping keyed on a stable serialization (kills duplicate export subrequests); RAG importance-rescale divide-by-zero guarded; AI-gateway token warn+document fallback.
  • + isolatedDeclarations fix: LUNORA_ATTR got an explicit Readonly<{…}> annotation so the packem DTS build (--isolatedDeclarations, CI's dist:check) passes — this gate is stricter than tsc --noEmit and was missed on the individual branch.

Verification (on the merged branch)

Full 48-package build:packages succeeds; tsc clean for runtime/do/server/ai/cli; tests: runtime 751, do 1248, server 445, ai 121, cli 689.

Merge-order note: contains changes to create-worker.ts (also touched by the x402 PR) and shard-do.ts (also touched by the geospatial PR) — land this PR first, then rebase those.

Generated by the /improve advisor workflow (audit → isolated-worktree Opus executors → tech-lead review). Plan text + review notes on improve/alpha-audit under plans/.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

@prisis, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 58 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: f4ca4b4a-20b2-483b-a15a-ef98d4567ccf

📥 Commits

Reviewing files that changed from the base of the PR and between 66d8e11 and 1cdcf1a.

⛔ Files ignored due to path filters (13)
  • api-snapshots/do.api.md is excluded by none and included by none
  • api-snapshots/runtime.api.md is excluded by none and included by none
  • packages/ai/__tests__/gateway.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/ai/__tests__/rag.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/do/__tests__/context-telemetry-cf-bridge.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/do/__tests__/shard-do.sampling.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/do/__tests__/tracing.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/runtime/__tests__/health.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/runtime/__tests__/observability-sinks.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/runtime/__tests__/otlp-export.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/runtime/__tests__/pipeline-log-reader.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/runtime/__tests__/sampling.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • shared/otlp.ts is excluded by none and included by none
📒 Files selected for processing (18)
  • apps/docs/src/content/docs/concepts/observability.mdx
  • apps/docs/src/content/docs/production-checklist.mdx
  • packages/ai/src/create-ai.ts
  • packages/ai/src/gateway.ts
  • packages/ai/src/rag/define-rag.ts
  • packages/cli/src/commands/logs/durable.ts
  • packages/cli/src/commands/logs/index.ts
  • packages/do/src/context-telemetry.ts
  • packages/do/src/database-telemetry.ts
  • packages/do/src/metric-history.ts
  • packages/do/src/shard-do.ts
  • packages/runtime/src/create-worker.ts
  • packages/runtime/src/health-routes.ts
  • packages/runtime/src/observability-sinks.ts
  • packages/runtime/src/observability.ts
  • packages/runtime/src/otlp-export.ts
  • packages/runtime/src/pipeline-log-reader.ts
  • packages/server/src/otel.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch advisor/wave15-observability

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@netlify

netlify Bot commented Jul 26, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit 1cdcf1a
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a65bc65b3f14300081507a6
😎 Deploy Preview https://deploy-preview-198--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@codspeed

codspeed Bot commented Jul 26, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 7.25%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 3 regressed benchmarks
✅ 235 untouched benchmarks
⏩ 2 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ in-batch: single IN(...) query + id->doc re-projection 795.1 µs 944.3 µs -15.79%
❌ count, no attributes 57.6 µs 65 µs -11.32%
❌ 1 shard × 1000 rows (single round-trip) 2.9 ms 3.2 ms -10.17%
⚡ baseline (Object.keys + toInternal + path spread per field) 71.9 µs 65.2 µs +10.31%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing advisor/wave15-observability (1cdcf1a) with alpha (0eab929)2

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on alpha (66d8e11) during the generation of this report, so 0eab929 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

prisis and others added 17 commits July 26, 2026 09:50
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ndaries

The durable R2 SQL log archive paged with an exclusive keyset cursor (ts <
cursor.ts) taken from an un-returned overflow row, dropping every row that
shared the overflow row's millisecond at each page boundary. Switch to an
inclusive resume (ts <= last-returned-row.ts) plus a per-boundary 'seen' hash
set carried on the cursor, so tied-ts rows are neither skipped nor duplicated.
A page dominated by a single ts grows its fetch window (bounded, 3 attempts) so
pagination still advances. The CLI --cursor now round-trips the full cursor as
an opaque token; Studio already passes it back opaquely.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every ctx.metrics.count/gauge/record was an unconditional durable SQLite
write on the request path, competing with app data for the shard's write
budget, with no off switch — unlike every other telemetry surface in this
delta. Add a `metricHistory` flag (default off) to the observability sink
options; gate the durable write in `recordMetric` before touching the raw
SQL handle, and thread `maxSeries`/`retentionBuckets` caps from the option
into metric-history.ts (constants stay the defaults). Enable the flag in the
getMetricHistory test's setup so it still exercises the persistence path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… span start

Two coupled fixes in the same `recordSpan` verdict path:

Step 2 — sampled-out dispatch no longer copies the whole span ring in every
`finally`. Held spans are buffered on the per-trace `traceSampling` entry
(bounded by MAX_HELD_SPANS_PER_TRACE) instead of read back from the shared
bounded ring, so a concurrent trace can no longer evict this trace's held
error spans before the flush reads them — which silently defeated
`alwaysSampleErrors`. `flushSampledOutTrace` drains that array and gains an
allocation-free short-circuit; the sampled-out flush path no longer calls
`this.spans.entries()`.

Step 3 — a span that ends after its dispatch settles no longer bypasses the
sampling verdict. `recordSpan` now takes the verdict snapshotted onto the span
when it opened (anchor.sampled, the createTracedFetch pattern) and, when the
live `traceSampling` entry is already gone, honours it: a late child of a
sampled-out trace is dropped rather than exported unconditionally. Self-anchored
subscription/alarm spans (snapshot sampled-in) still export.

The BridgeSpan lifetime bound (settling an abandoned span so its suspended
ctx.trace body can't pin the closure) is deferred per the plan's escape hatch:
the clean route is a dispatch-teardown hook and the LunoraTraceContext surface
exposes none, while a timer is the mechanism the DO profile avoids. The
correctness half (no orphan export) is done; documented on BridgeSpan.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DO emitted `lunora.error.type`/`lunora.error.message` while the worker
exporter emitted `error.type`, so a collector query on either key silently
matched one source and dropped the other. Introduce a frozen `LUNORA_ATTR`
record in shared/otlp.ts as the single source of truth for reserved keys and
route every emit path through it.

WIRE-FORMAT CHANGE: the DO's error span attributes converge on the OTel-standard
`error.type` and `error.message` (were `lunora.error.type`/`lunora.error.message`).
Dashboards/queries filtering the old DO keys must switch to the OTel keys; no
bundled Studio query referenced them. The lunora.* dispatch keys
(function_path/ok/shard_key/user_id/duration_ms) are unchanged in value.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gzip threshold compared `json.length` (UTF-16 code units) against
OTLP_GZIP_THRESHOLD, so non-ASCII log/span text (common in error.message) could
sit well over the threshold in bytes yet skip gzip. Measure the encoded byte
length instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dentity

resourceAttributesFor memoizes a merged resource bag per request context, so two
requests in one flush window with byte-identical resource bags produced two
distinct map keys, two OTLP envelopes, and two fetch calls — the subrequest
multiplication the batcher exists to prevent. Key the groups on
stableStringify(resource) so identical bags share one envelope / one fetch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
chunk.importance can be 0 (a documented, validated value); when it is indexed,
the text-store hydrate path computed score / chunk.importance = 0 / 0 = NaN,
corrupting the whole retrieval ordering rather than the single chunk. Guard the
divide so importance 0 yields a finite score, and fix the misleading
'always 1 here' comment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ding

An operator who set LUNORA_AI_GATEWAY_TOKEN for the Workers AI binding path got
every ctx.ai.model(...) call failing at runtime against an authenticated gateway
while the token var read as 'configured': Cloudflare's native binding `gateway`
option (GatewayOptions) has no authorization field, so resolveGatewayOption
discarded the token. The token cannot be threaded on this path (verified against
workers-ai-provider@4 / @cloudflare/workers-types GatewayOptions), so warn once
per isolate from resolveAiGateway on the workers-ai-binding consumer and document
the limitation on AI_GATEWAY_TOKEN_ENV. The BYO-provider path still sends the
token as cf-aig-authorization.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…seTally

The JSDoc describing formatTally sat after createDatabaseTally's return
statement, documenting nothing; move it above formatTally where it belongs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three observability review findings:

- Readiness probe (`/ready`) no longer inherits the 5s public-posture
  default cache. A k8s / load-balancer readiness poll must observe a
  dependency going down on the next poll, not up to 5s later, so the
  readiness default is now 0 (uncached) in both postures; an explicit
  `cacheTtlMs` still opts it into caching. The aggregate probe keeps its
  5s public default.
- A custom probe name lacking a `kind:` prefix (e.g. an operator-supplied
  `acme-prod-billing`) is now redacted to a generic `probe` label in the
  public posture instead of leaking verbatim to an unauthenticated caller.
  The internal DO probe is renamed `durable-object:default` so it keeps
  its safe `durable-object` public label via the same colon rule. Admin
  posture still returns full names.
- Stop passing the now-dead `sampling` argument at the two settled-decision
  `emitRpcEvent` call sites; `emitRpcEvent` ignores it whenever a
  `decision` is present. No behavior change.

Also extracts `redactCheckName` / `overallStatus` from `buildBody` to keep
its cognitive complexity within the lint budget.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`shard-do.ts`'s `TelemetrySink.metricHistory` now references the exported
`MetricHistoryOptions` (same package) rather than re-declaring the
`{ maxSeries?; retentionBuckets? }` literal, and the dead conditional-spread
that stripped `undefined` keys before `recordMetricHistory` (which already
coalesces every field with `?? DEFAULT`) collapses to a straight pass-through.
The runtime-side copy in `observability.ts` keeps the inline literal — importing
the type would couple this public type to `@lunora/do` — with a keep-in-sync note.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The durable path prints an opaque base64url cursor token, not a bare `ts`.
Update the `--cursor` option help to say it takes the token printed by the
previous page (bare epoch-millis still accepted for back-compat).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Green up the CI lint gates the branch was red on, independent of the fixes:

- eslint autofixes on wave-15 files (`otlp-export.ts` prefer-destructuring,
  `observability-sinks.test.ts` arrow-body-style).
- prettier reflow of `sampling.test.ts`.
- refresh `do`/`runtime` api snapshots, which had drifted from already-committed
  wave-15 code (`recordSpan` sampled-snapshot param, `ObservabilitySink.metricHistory`,
  `PipelineLogCursor.seen`, `emitRpcEvent` decision param) plus this branch's
  `TelemetrySink.metricHistory` → `MetricHistoryOptions` change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@prisis
prisis force-pushed the advisor/wave15-observability branch from 181aef0 to 1cdcf1a Compare July 26, 2026 07:50
@prisis
prisis merged commit c79937c into alpha Jul 26, 2026
33 of 34 checks passed
@prisis
prisis deleted the advisor/wave15-observability branch July 26, 2026 07:52
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — 1cdcf1ae Deployed Jul 26, 2026 by prisis via Benchmarks #995
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant