Repository navigation
feat(observability)!: sampling coherence, health, log-archive, DO telemetry + OTLP (Wave 15: 179,180,181,186,190) - #198
Conversation
|
Warning Review limit reached
Next review available in: 58 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (13)
📒 Files selected for processing (18)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for following the naming conventions! 🙏 |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Merging this PR will degrade performance by 7.25%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | in-batch: single IN(...) query + id->doc re-projection |
795.1 µs | 944.3 µs | -15.79% |
| ❌ | count, no attributes |
57.6 µs | 65 µs | -11.32% |
| ❌ | 1 shard × 1000 rows (single round-trip) |
2.9 ms | 3.2 ms | -10.17% |
| ⚡ | baseline (Object.keys + toInternal + path spread per field) |
71.9 µs | 65.2 µs | +10.31% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing advisor/wave15-observability (1cdcf1a) with alpha (0eab929)2
Footnotes
-
2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
alpha(66d8e11) during the generation of this report, so 0eab929 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ndaries The durable R2 SQL log archive paged with an exclusive keyset cursor (ts < cursor.ts) taken from an un-returned overflow row, dropping every row that shared the overflow row's millisecond at each page boundary. Switch to an inclusive resume (ts <= last-returned-row.ts) plus a per-boundary 'seen' hash set carried on the cursor, so tied-ts rows are neither skipped nor duplicated. A page dominated by a single ts grows its fetch window (bounded, 3 attempts) so pagination still advances. The CLI --cursor now round-trips the full cursor as an opaque token; Studio already passes it back opaquely. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every ctx.metrics.count/gauge/record was an unconditional durable SQLite write on the request path, competing with app data for the shard's write budget, with no off switch — unlike every other telemetry surface in this delta. Add a `metricHistory` flag (default off) to the observability sink options; gate the durable write in `recordMetric` before touching the raw SQL handle, and thread `maxSeries`/`retentionBuckets` caps from the option into metric-history.ts (constants stay the defaults). Enable the flag in the getMetricHistory test's setup so it still exercises the persistence path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… span start Two coupled fixes in the same `recordSpan` verdict path: Step 2 — sampled-out dispatch no longer copies the whole span ring in every `finally`. Held spans are buffered on the per-trace `traceSampling` entry (bounded by MAX_HELD_SPANS_PER_TRACE) instead of read back from the shared bounded ring, so a concurrent trace can no longer evict this trace's held error spans before the flush reads them — which silently defeated `alwaysSampleErrors`. `flushSampledOutTrace` drains that array and gains an allocation-free short-circuit; the sampled-out flush path no longer calls `this.spans.entries()`. Step 3 — a span that ends after its dispatch settles no longer bypasses the sampling verdict. `recordSpan` now takes the verdict snapshotted onto the span when it opened (anchor.sampled, the createTracedFetch pattern) and, when the live `traceSampling` entry is already gone, honours it: a late child of a sampled-out trace is dropped rather than exported unconditionally. Self-anchored subscription/alarm spans (snapshot sampled-in) still export. The BridgeSpan lifetime bound (settling an abandoned span so its suspended ctx.trace body can't pin the closure) is deferred per the plan's escape hatch: the clean route is a dispatch-teardown hook and the LunoraTraceContext surface exposes none, while a timer is the mechanism the DO profile avoids. The correctness half (no orphan export) is done; documented on BridgeSpan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DO emitted `lunora.error.type`/`lunora.error.message` while the worker exporter emitted `error.type`, so a collector query on either key silently matched one source and dropped the other. Introduce a frozen `LUNORA_ATTR` record in shared/otlp.ts as the single source of truth for reserved keys and route every emit path through it. WIRE-FORMAT CHANGE: the DO's error span attributes converge on the OTel-standard `error.type` and `error.message` (were `lunora.error.type`/`lunora.error.message`). Dashboards/queries filtering the old DO keys must switch to the OTel keys; no bundled Studio query referenced them. The lunora.* dispatch keys (function_path/ok/shard_key/user_id/duration_ms) are unchanged in value. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gzip threshold compared `json.length` (UTF-16 code units) against OTLP_GZIP_THRESHOLD, so non-ASCII log/span text (common in error.message) could sit well over the threshold in bytes yet skip gzip. Measure the encoded byte length instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dentity resourceAttributesFor memoizes a merged resource bag per request context, so two requests in one flush window with byte-identical resource bags produced two distinct map keys, two OTLP envelopes, and two fetch calls — the subrequest multiplication the batcher exists to prevent. Key the groups on stableStringify(resource) so identical bags share one envelope / one fetch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
chunk.importance can be 0 (a documented, validated value); when it is indexed, the text-store hydrate path computed score / chunk.importance = 0 / 0 = NaN, corrupting the whole retrieval ordering rather than the single chunk. Guard the divide so importance 0 yields a finite score, and fix the misleading 'always 1 here' comment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ding An operator who set LUNORA_AI_GATEWAY_TOKEN for the Workers AI binding path got every ctx.ai.model(...) call failing at runtime against an authenticated gateway while the token var read as 'configured': Cloudflare's native binding `gateway` option (GatewayOptions) has no authorization field, so resolveGatewayOption discarded the token. The token cannot be threaded on this path (verified against workers-ai-provider@4 / @cloudflare/workers-types GatewayOptions), so warn once per isolate from resolveAiGateway on the workers-ai-binding consumer and document the limitation on AI_GATEWAY_TOKEN_ENV. The BYO-provider path still sends the token as cf-aig-authorization. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…seTally The JSDoc describing formatTally sat after createDatabaseTally's return statement, documenting nothing; move it above formatTally where it belongs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three observability review findings: - Readiness probe (`/ready`) no longer inherits the 5s public-posture default cache. A k8s / load-balancer readiness poll must observe a dependency going down on the next poll, not up to 5s later, so the readiness default is now 0 (uncached) in both postures; an explicit `cacheTtlMs` still opts it into caching. The aggregate probe keeps its 5s public default. - A custom probe name lacking a `kind:` prefix (e.g. an operator-supplied `acme-prod-billing`) is now redacted to a generic `probe` label in the public posture instead of leaking verbatim to an unauthenticated caller. The internal DO probe is renamed `durable-object:default` so it keeps its safe `durable-object` public label via the same colon rule. Admin posture still returns full names. - Stop passing the now-dead `sampling` argument at the two settled-decision `emitRpcEvent` call sites; `emitRpcEvent` ignores it whenever a `decision` is present. No behavior change. Also extracts `redactCheckName` / `overallStatus` from `buildBody` to keep its cognitive complexity within the lint budget. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`shard-do.ts`'s `TelemetrySink.metricHistory` now references the exported
`MetricHistoryOptions` (same package) rather than re-declaring the
`{ maxSeries?; retentionBuckets? }` literal, and the dead conditional-spread
that stripped `undefined` keys before `recordMetricHistory` (which already
coalesces every field with `?? DEFAULT`) collapses to a straight pass-through.
The runtime-side copy in `observability.ts` keeps the inline literal — importing
the type would couple this public type to `@lunora/do` — with a keep-in-sync note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The durable path prints an opaque base64url cursor token, not a bare `ts`. Update the `--cursor` option help to say it takes the token printed by the previous page (bare epoch-millis still accepted for back-compat). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Green up the CI lint gates the branch was red on, independent of the fixes: - eslint autofixes on wave-15 files (`otlp-export.ts` prefer-destructuring, `observability-sinks.test.ts` arrow-body-style). - prettier reflow of `sampling.test.ts`. - refresh `do`/`runtime` api snapshots, which had drifted from already-committed wave-15 code (`recordSpan` sampled-snapshot param, `ObservabilitySink.metricHistory`, `PipelineLogCursor.seen`, `emitRpcEvent` decision param) plus this branch's `TelemetrySink.metricHistory` → `MetricHistoryOptions` change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
181aef0 to
1cdcf1a
Compare
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Wave 15 — observability, sampling & OTLP correctness (plans 179, 180, 181, 186, 190)
Five reviewed plans in the telemetry/runtime/DO surface, merged into one branch (all touch
create-worker.ts/shard-do.ts/observability.ts— conflicts resolved and the combined suite re-run).traceparentand the exported SERVER span can never disagree (re-fixes the orphan-span class from fix(do): propagate the inbound sampled bit instead of hardcoding it #191). +agreement tests./_lunora/healthhardening: the public posture no longer echoes operator binding names (recon leak vs the module's own invariant), andcacheTtlMsis now real (was a documented no-op that left the unauthenticated endpoint running live DO/D1 probes per request).ts <=resume + last-returned-row cursor +seen-hash dedup + degenerate-page grow. Completeness test proves union-of-pages == full result.metricHistoryis now an opt-out (default off — it was an unconditional durable SQLite write perctx.metrics.*); sampled-out held spans live on the per-trace entry (no whole-ring copy per dispatch; eviction race closed); a span ending after its sampled-out dispatch tore down no longer exports unconditionally (verdict snapshot).LUNORA_ATTR(wire-format change — seefix(otlp)!); gzip decided by byte length (not UTF-16 units); resource grouping keyed on a stable serialization (kills duplicate export subrequests); RAG importance-rescale divide-by-zero guarded; AI-gateway token warn+document fallback.LUNORA_ATTRgot an explicitReadonly<{…}>annotation so the packem DTS build (--isolatedDeclarations, CI'sdist:check) passes — this gate is stricter thantsc --noEmitand was missed on the individual branch.Verification (on the merged branch)
Full 48-package
build:packagessucceeds;tscclean for runtime/do/server/ai/cli; tests: runtime 751, do 1248, server 445, ai 121, cli 689.Generated by the
/improveadvisor workflow (audit → isolated-worktree Opus executors → tech-lead review). Plan text + review notes onimprove/alpha-auditunderplans/.🤖 Generated with Claude Code