Repository navigation
feat(otel): close otel-cf-workers gaps and add wide events - #189
Conversation
Three additions the exporter needs, on top of the resource work already on alpha. shared/otlp-batch.ts is a minimal BatchSpanProcessor reduced to what a Workers isolate needs: no background thread, no unbounded queue, no retry state machine. A Worker is capped at 50/1000 subrequests per invocation, so one fetch per span is a correctness problem, not only an efficiency one. The wrapResource* envelopes now take one item or many, which is what lets a whole invocation ship in a single request. SpanEvent gains kind, events and links; SpanHandle gains addEvent, addLink, recordException and spanContext. A trace where everything is INTERNAL yields no service map, and a HANDLED exception previously had nowhere to go — nothing re-throws it for a top-level handler to log. LogEvent gains eventName for OTel's Events API, so a structured event is distinguishable from a human-readable log line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three gaps, one theme: make a request observable without making the developer write a log line per fact. ctx.span is a handle on the span the dispatch ALREADY emits. Attributes accumulate across the request and export as one OTel Event record (lunora.dispatch) — flat cost however much you attach. It is a log record rather than span attributes because the DO's root span shares its span id with the runtime's SERVER span, so exporting both would put two partial spans with identical ids on the wire. ctx.log.event adds the discrete-event counterpart. ctx.fetch is instrumented: a CLIENT span plus a traceparent naming the span it actually records. The id is minted before the request precisely so the header cannot point at a span that never existed. ctx.db instrumentation is tiered so it can default to on: "summary" folds flat-cost counters onto the wide event, "spans" gives the full waterfall capped per dispatch (reporting the truncation rather than hiding it), "off" returns the db untouched. alarm() and webSocketMessage() get the same trigger treatment the worker's queue/cron entries do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Types for the wide-event API. The doc comments carry the rule
worth internalising: log.* for narration you would read while
debugging, log.event for anything you would put on a dashboard,
and ctx.span for facts about the request itself.
ctx.trace's third argument now also accepts SpanOptions (kind,
links). It is read as options only when EVERY key is one of
attributes/kind/links, so an ordinary attribute bag is unaffected
and { attributes: { kind: ... } } resolves the one ambiguity.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Instrumentation that cannot be asserted on does not get maintained. ctx.span in the harness now records rather than discards, and t.wideEvent() exposes what a handler attached, so "did we record the right thing?" is answerable in a test instead of only after a deploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
buildCtx now resolves the telemetry sink and trace anchor BEFORE ctx.db, because the database is wrapped in auto-instrumentation that needs both plus the dispatch's wide-event handle. ctx.trace and ctx.span share one anchor so they land in the same trace. Golden fixture regenerated; examples/expo picked up the same change from a test run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
otlpSink buffered nothing: every span, log line and metric was its own fetch, so a well-instrumented handler could spend the invocation's whole subrequest budget on telemetry. It now buffers and ships one request per signal, flushed at the invocation boundary via the new ObservabilitySink.flush. Buffering is also what makes real tail sampling possible: by flush time a trace has settled, so tailSampler can keep it on "anything failed or ran slow" — and a dropped trace takes its correlated log records with it rather than leaving orphans. The encoders in otlp-export move to returning bare objects so a batch can share one envelope, and a flush window is grouped by RESOURCE before wrapping: with detectResources on, two requests in one window can carry different host attributes and an envelope has exactly one resource, so mixing them would mis-attribute the batch. Also adds postProcessor (the redaction seam, failing open so a bad rule shows as un-redacted data rather than as silence), span kind/events/links encoding, and LogRecord.eventName. Builds on alpha's resource work rather than duplicating it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Warning Review limit reached
Next review available in: 32 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (5)
📒 Files selected for processing (1)
WalkthroughThis PR expands observability with span events, links, exceptions, database and fetch instrumentation, trigger tracing, buffered OTLP export, an OpenTelemetry tracer bridge, wide-event testing support, and bundled OpenTelemetry licensing. ChangesObservability platform
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Handler
participant ShardDO
participant ContextTelemetry
participant OTLPBatcher
participant OTLPExporter
Handler->>ShardDO: invoke dispatch or trigger
ShardDO->>ContextTelemetry: create trace and span collector
Handler->>ContextTelemetry: record events, links, fetch, or database telemetry
ContextTelemetry->>OTLPBatcher: buffer correlated signals
ShardDO->>OTLPBatcher: flush at invocation boundary
OTLPBatcher->>OTLPExporter: encode and send OTLP payloads
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for following the naming conventions! 🙏 |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
A library instrumented against @opentelemetry/api cannot see Lunora's hand-rolled trace: its spans go to the global provider, which in a Worker is the no-op one, and vanish. createOtelTracer(ctx) returns a real OTel Tracer whose spans ARE ctx.trace spans. The two lifetime models are bridged by suspending the ctx.trace body on a promise that end() resolves, so Lunora measures the real interval and no part of the runtime needs a special case. There is deliberately no ambient context: tracking a current span across awaits needs AsyncLocalStorage, which DOs avoid. Spans parent to the dispatch and the tracer is passed explicitly (the AI SDK's experimental_telemetry.tracer) — flatter than a full SDK, never wrong, never lost. Shipped as the @lunora/server/otel subpath rather than its own package: 400 lines do not warrant a release stream, and an OPTIONAL peer dependency is how this repo already handles third-party integrations (@lunora/ai, @lunora/payment, @lunora/hyperdrive). Nothing installs @opentelemetry/api unless it imports the subpath, and the separate packem entry keeps it out of the main bundle. It is a peer rather than a dependency because the API keeps a module-global registry, so a second copy would silently split it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/runtime/src/observability-sinks.ts (1)
992-1032: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
combineSinksdoesn't forward the newflushhook — buffered signals from a wrapped batching sink can be silently lost.
fanOut's method union ("onLog" | "onMetric" | "onRpc" | "onSpan") and the returned sink object never referenceflush. SinceflushSinktreats a missingflushas a no-op (if (!sink?.flush) return;), any batching sink (e.g. this file's ownotlpSink, which now batches by default) wrapped throughcombineSinks— the pattern this very module's docs recommend ("wired behind {@linkcombineSinks} alongside a network sink") — never receives the invocation-boundary flush call. Buffered telemetry then depends entirely on the batcher's internalmaxDelayMs/maxItemstimers, which can miss the isolate-freeze window the newflushhook exists specifically to guard against, per this file's own docs onObservabilitySink.flush.🐛 Proposed fix to forward `flush` to every child sink
return { + flush: (context?: ObservabilitySinkContext) => { + for (const sink of sinks) { + if (!sink.flush) { + continue; + } + + try { + sink.flush(context); + } catch { + // Isolate failures so one bad sink doesn't starve the rest. + } + } + }, onLog: (event: LogEvent, context?: ObservabilitySinkContext) => { fanOut("onLog", event, context); },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/runtime/src/observability-sinks.ts` around lines 992 - 1032, Update combineSinks and its fanOut method to include the ObservabilitySink flush hook, forwarding each flush invocation to every child sink that implements flush while isolating individual failures consistently with the existing signal handlers. Ensure the returned combined sink exposes flush so flushSink invokes wrapped batching sinks.
🧹 Nitpick comments (2)
packages/runtime/src/create-worker.ts (1)
3898-3904: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueExtract the
waitUntilsink-context construction.The same
context.waitUntil ? { waitUntil: (promise) => { context.waitUntil?.(promise); } } : undefinedshape now appears ininstrumentTriggerand in thefetchfinally. A tinysinkContextFor(context)helper keeps the two boundaries from drifting.Also applies to: 4316-4330
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/runtime/src/create-worker.ts` around lines 3898 - 3904, Extract the repeated conditional sink-context construction into a shared sinkContextFor(context) helper, preserving the existing undefined behavior when context.waitUntil is absent and forwarding promises to context.waitUntil when present. Replace the inline constructions in instrumentTrigger and the fetch finally path with this helper so both boundaries use the same implementation.packages/runtime/src/otlp-export.ts (1)
18-24: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRedundant
parentSpanIdassignment — spread (line 89) and theifblock (lines 99-101) do the same thing.Both set
span.parentSpanIdfromevent.parentSpanIdunder the same condition; theifblock is now dead weight left over after the new conditional spread was added.As per path instructions, "Follow DRY principles."♻️ Proposed cleanup
const span: Record<string, unknown> = { attributes, endTimeUnixNano: otlpUnixNano(endMs), // SPAN_KIND_SERVER — a dispatched RPC is server-side request handling. kind: OTLP_SPAN_KIND.server, name: event.functionPath, // Set only when the worker joined an upstream trace; omitted otherwise, so // a self-originated dispatch stays the root rather than dangling off a // parent that was never exported. ...(event.parentSpanId === undefined ? {} : { parentSpanId: event.parentSpanId }), // Reuse the dispatch's trace context when the runtime set it (so this span // shares the id it propagated as `traceparent`); else mint fresh ids. spanId: event.spanId ?? otlpRandomHex(8), startTimeUnixNano: otlpUnixNano(endMs - event.durationMs), // STATUS_CODE_OK (1) / STATUS_CODE_ERROR (2). status: event.ok ? { code: 1 } : { code: 2, message: event.error?.message ?? "" }, traceId: event.traceId ?? otlpRandomHex(16), }; - if (event.parentSpanId !== undefined) { - span.parentSpanId = event.parentSpanId; - } - if (event.traceFlags !== undefined) { span.flags = event.traceFlags; }Also applies to: 80-101, 119-120
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/runtime/src/otlp-export.ts` around lines 18 - 24, Remove the redundant parentSpanId assignment from the conditional block in otlpTraceBody, leaving the existing conditional spread as the sole assignment from event.parentSpanId. Preserve the surrounding span construction and all other conditional fields unchanged.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/do/src/database-telemetry.ts`:
- Around line 289-307: Change the telemetry flow around publishTally and the
dispatch-boundary handling so summary-mode database tallies do not materialize
deps.span or count as a handler-wide event on every ctx.db call. Publish the
accumulated tally once at the dispatch boundary, or route summary attributes
through a non-wide-event channel, while preserving spans-mode emission and
ensuring shard-do.ts’s root-span gate only fires when the handler actually
records a wide event.
In `@packages/do/src/shard-do.ts`:
- Around line 205-215: Reposition the detached JSDoc blocks in
packages/do/src/shard-do.ts: move the block currently spanning lines 192–204
immediately above fuseCloudflareTraces, move the 4641–4658 block immediately
above makeDispatchSpan, and move the 4677–4689 block immediately above
makeFetch. Keep each declaration’s existing behavior unchanged and ensure no
documentation block remains separated from the member it describes.
- Around line 1950-1967: Update the dispatchSpans keying from traceId-only to
the anchor span identity, such as a traceId/rootSpanId combination, so
concurrent dispatches sharing a trace use separate wide-event collectors. Apply
the same key derivation consistently in makeDispatchSpan, the fetch finally
cleanup, withTriggerTrace, and recordDispatchRootSpan, preserving per-dispatch
cleanup and root-span gating.
In `@packages/runtime/src/create-worker.ts`:
- Around line 3906-3915: Update the trigger-event emissions in the run wrapper
around emitRpcEvent so they are exported independently of head-trace sampling,
or use a dedicated trigger sampling ratio. Ensure both success and error events
remain visible for low-frequency triggers while preserving their existing event
payloads and error propagation.
In `@packages/server/src/otel.ts`:
- Around line 242-246: Update Span.recordException to normalize non-Error
Exception objects before passing them to handle.recordException, preserving
string and Error inputs while converting object values to a message-bearing
exception using their message and available fields.
In `@packages/testing/src/harness.ts`:
- Around line 334-339: Update the recordException callback in SpanHandle to
include the documented exception.stacktrace field when recording the "exception"
event. Preserve the existing Error checks and type/message behavior, using the
Error stack when available and an appropriate fallback for non-Error values so
harness events match production instrumentation.
---
Outside diff comments:
In `@packages/runtime/src/observability-sinks.ts`:
- Around line 992-1032: Update combineSinks and its fanOut method to include the
ObservabilitySink flush hook, forwarding each flush invocation to every child
sink that implements flush while isolating individual failures consistently with
the existing signal handlers. Ensure the returned combined sink exposes flush so
flushSink invokes wrapped batching sinks.
---
Nitpick comments:
In `@packages/runtime/src/create-worker.ts`:
- Around line 3898-3904: Extract the repeated conditional sink-context
construction into a shared sinkContextFor(context) helper, preserving the
existing undefined behavior when context.waitUntil is absent and forwarding
promises to context.waitUntil when present. Replace the inline constructions in
instrumentTrigger and the fetch finally path with this helper so both boundaries
use the same implementation.
In `@packages/runtime/src/otlp-export.ts`:
- Around line 18-24: Remove the redundant parentSpanId assignment from the
conditional block in otlpTraceBody, leaving the existing conditional spread as
the sole assignment from event.parentSpanId. Preserve the surrounding span
construction and all other conditional fields unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 53099413-a6aa-4942-85eb-e8c5121f1715
⛔ Files ignored due to path filters (16)
examples/expo/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by nonepackages/codegen/__tests__/fixtures/simple/expected/_generated/shard.tsis excluded by!**/_generated/**,!**/__tests__/**and included bypackages/**packages/do/__tests__/database-telemetry.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/do/__tests__/traced-fetch.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/observability-sinks.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/server/__tests__/context.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/server/__tests__/functions-v-from.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/server/__tests__/functions.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/server/__tests__/otel.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/testing/__tests__/wide-event.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**pnpm-lock.yamlis excluded by!**/pnpm-lock.yamland included by nonepnpm-workspace.yamlis excluded by none and included by noneshared/log-event.tsis excluded by none and included by noneshared/otlp-batch.tsis excluded by none and included by noneshared/otlp.tsis excluded by none and included by noneshared/span-event.tsis excluded by none and included by none
📒 Files selected for processing (16)
packages/cloudflare-access/LICENSE.mdpackages/codegen/src/emit.tspackages/do/src/context-telemetry.tspackages/do/src/database-telemetry.tspackages/do/src/shard-do.tspackages/replica/LICENSE.mdpackages/runtime/src/create-worker.tspackages/runtime/src/observability-sinks.tspackages/runtime/src/observability.tspackages/runtime/src/otlp-export.tspackages/server/LICENSE.mdpackages/server/package.jsonpackages/server/src/index.tspackages/server/src/otel.tspackages/server/src/types.tspackages/testing/src/harness.ts
Merging this PR will degrade performance by 18.54%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | deep mixed stack |
618.7 µs | 759.5 µs | -18.54% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing feat/otel-parity (6c1e4b1) with alpha (49b10ad)2
Footnotes
-
2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
alpha(aa7abe6) during the generation of this report, so 49b10ad was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
combineSinks did not forward the new flush hook, so any batching sink wrapped through it — the pairing this module's own docs recommend — never received the invocation-boundary flush, and its buffer died with the isolate. fanOut now takes an argument list, because flush(context) puts the context first while the four on*(event, context) hooks put it second. Covered by a test. dispatchSpans was keyed by traceId alone. resolveTraceAnchor takes traceId from the inbound traceparent, so two concurrent RPCs under one client trace shared an entry: their wide events merged and whichever finished first deleted the other's. Keyed by trace id AND root span id now. traceSampling keeping traceId is still right — a sampling verdict is per-trace, a wide event is per-dispatch. Summary-mode db counters wrote through ctx.span, which materializes the wide-event collector. Since summary is the default, nearly every request then looked like the handler had recorded a wide event, firing the root-span gate and exporting a lunora.dispatch record for handlers that instrumented nothing — the opposite of what the gate exists for. The tally now lands on a separate slot that enriches a root span being recorded anyway, but never causes one. Trigger events no longer ride the head-sampling ratio: a ratio tuned for request traffic hid most fires of an hourly cron, which is the blind spot the wrapper was added to close. Also: OTel Exception's object form keeps its message instead of stringifying to [object Object]; dropped a dead `ok` write CodeQL flagged (it was exactly `error === undefined`); removed a duplicated parentSpanId assignment; extracted sinkContextFor so both invocation boundaries share one shape; reattached four JSDoc blocks that member reordering had orphaned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/runtime/src/observability-sinks.ts (1)
42-160: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winDo not export the unprocessed signal when redaction fails.
The fail-open
postProcesspath can send unredacted telemetry after a redaction hook throws. This is especially risky becauseObservabilityEvent.error.messagemay contain user input, and third-party sinks are expected to scrub it. Drop the affected signal or make unredacted fallback an explicit opt-in with a diagnostic metric.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/runtime/src/observability-sinks.ts` around lines 42 - 160, Update postProcess and encodeSignal so a post-processor exception never returns or encodes the original unredacted event. Drop the affected signal by returning undefined, preserving normal hook-returned events and explicit hook drops; do not add an unredacted fallback path.
🧹 Nitpick comments (1)
packages/runtime/src/observability-sinks.ts (1)
42-160: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy liftMake tail-sampler failures observable and bounded.
When
tailSamplerthrows, the implementation keeps the entire trace, silently bypassing the configured sampling policy. Repeated failures can substantially increase export volume and cost. Emit a rate-limited diagnostic/metric and document whether fail-open is intentional; consider a configurable fail-closed policy when sampling is a volume-control guarantee.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/runtime/src/observability-sinks.ts` around lines 42 - 160, Update the catch block in applyTailSampler to emit a rate-limited diagnostic or metric when tailSampler throws, preventing repeated failures from generating unbounded telemetry. Document that the current behavior intentionally fails open and retains the trace; if the existing configuration supports sampling as a volume-control guarantee, expose and honor a configurable fail-closed policy.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@packages/runtime/src/observability-sinks.ts`:
- Around line 42-160: Update postProcess and encodeSignal so a post-processor
exception never returns or encodes the original unredacted event. Drop the
affected signal by returning undefined, preserving normal hook-returned events
and explicit hook drops; do not add an unredacted fallback path.
---
Nitpick comments:
In `@packages/runtime/src/observability-sinks.ts`:
- Around line 42-160: Update the catch block in applyTailSampler to emit a
rate-limited diagnostic or metric when tailSampler throws, preventing repeated
failures from generating unbounded telemetry. Document that the current behavior
intentionally fails open and retains the trace; if the existing configuration
supports sampling as a volume-control guarantee, expose and honor a configurable
fail-closed policy.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 29375b51-b502-4ef9-8e06-0c8372e937b9
⛔ Files ignored due to path filters (5)
examples/expo/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by nonepackages/codegen/__tests__/fixtures/simple/expected/_generated/shard.tsis excluded by!**/_generated/**,!**/__tests__/**and included bypackages/**packages/do/__tests__/database-telemetry.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/observability-sinks.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**pnpm-lock.yamlis excluded by!**/pnpm-lock.yamland included by none
📒 Files selected for processing (9)
packages/codegen/src/emit.tspackages/do/src/context-telemetry.tspackages/do/src/database-telemetry.tspackages/do/src/shard-do.tspackages/runtime/src/create-worker.tspackages/runtime/src/observability-sinks.tspackages/runtime/src/otlp-export.tspackages/server/package.jsonpackages/server/src/otel.ts
💤 Files with no reviewable changes (1)
- packages/runtime/src/otlp-export.ts
🚧 Files skipped from review as they are similar to previous changes (7)
- packages/server/package.json
- packages/runtime/src/create-worker.ts
- packages/codegen/src/emit.ts
- packages/do/src/database-telemetry.ts
- packages/do/src/shard-do.ts
- packages/server/src/otel.ts
- packages/do/src/context-telemetry.ts
SpanHandle.recordException promises exception.type, exception.message AND exception.stacktrace. The harness recorder omitted the last one, so a test asserting on a wide event's exception attributes passed against a narrower record than the one production actually ships — which defeats the point of making the instrumentation assertable. The remaining divergence is documented inline: production derives exception.type via toErrorType, preferring a LunoraError's stable code, and that helper is not exported from @lunora/do. For a plain Error — the common case in a test — the two agree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Addressed all review findings. Summary of what changed and why: 🔴 🟠 **🟠 🟡 Head sampling silenced low-frequency triggers — correct. Trigger events no longer ride the ratio at all; volume is inherently low and a ratio tuned for request traffic hid most fires of an hourly cron, which is the blind spot the wrapper exists to close. 🟡 OTel 🟡 Detached JSDoc blocks — correct, all four reattached (member reordering for CodeQL: dead Nitpicks — both applied: extracted One more found while reviewing your harness comment, which was also right: the harness Verification: repo-wide |
Redaction now fails CLOSED. postProcess previously returned the event unmodified when a hook threw, on the reasoning that visible un-redacted data beats silent absence. That is right for observability in general and wrong for this hook: postProcessor exists to strip PII before telemetry leaves for a third-party collector, and error.message routinely carries user input. A dropped span is a monitoring gap you notice; a leaked payload is an incident you cannot retract. The test that asserted the old behaviour is inverted. Two benchmark regressions CodSpeed caught, both self-inflicted: broadcastWhisper to 128 members (-31%) — webSocketMessage was wrapped in withTriggerTrace, minting a trace anchor per frame (two crypto draws plus hex encoding) on an unbounded-frequency path. The wrapper is removed there. The trade was poor anyway: a frame's work is a subscription re-evaluation already attributable to the RPC that wrote the data. alarm keeps the wrapper — low frequency, and genuinely un-attributable background work, which was the blind spot worth closing. rankPage hydration (-17%) — two causes. The db tally was published per call, rebuilding a fields object and doing a map lookup on every query; the shard now reads a plain counter object once at the dispatch boundary, so a query pays only integer increments. And both makeDispatchSpan and the tally touched dispatchSpans eagerly on every buildCtx even with no sink configured; both now bail when nothing is collecting, so a deployment that exports no telemetry pays nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Correcting my earlier claim: I had not addressed everything. The 7 inline threads were fixed and are resolved, but I missed two later signals. Both are now fixed in 🟠 Security: redaction failed openThe third review flagged that ❌ Two CodSpeed regressions, both self-inflictedI hadn't looked at the performance report at all. Both regressions were mine:
Please re-run CodSpeed to confirm; local wall-clock isn't comparable to its simulated instruction counts, so I'm not claiming the numbers, only that the per-call and per-frame work is gone. Verification: repo-wide |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/do/src/database-telemetry.ts`:
- Around line 299-310: Move the documentation block currently placed after the
return statement in createDatabaseTally to immediately above the formatTally
declaration, so it documents formatTally rather than either declaration
indirectly. Keep the tally initialization and documentation text unchanged.
In `@packages/do/src/shard-do.ts`:
- Around line 5323-5327: Update the collected-building logic near
databaseAttributes so databaseAttributes are preserved even when wide.collector
is undefined; create an empty collection containing the database attributes
whenever a tally exists, while retaining existing collector attributes and merge
behavior when a collector is present.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 4052e383-4c61-4838-b6e8-6a7245d46cac
⛔ Files ignored due to path filters (4)
packages/do/__tests__/database-telemetry.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/observability-sinks.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/testing/__tests__/wide-event.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**pnpm-lock.yamlis excluded by!**/pnpm-lock.yamland included by none
📒 Files selected for processing (7)
packages/codegen/src/emit.tspackages/do/src/database-telemetry.tspackages/do/src/shard-do.tspackages/runtime/src/observability-sinks.tspackages/server/src/index.tspackages/server/src/types.tspackages/testing/src/harness.ts
🚧 Files skipped from review as they are similar to previous changes (4)
- packages/server/src/index.ts
- packages/codegen/src/emit.ts
- packages/server/src/types.ts
- packages/testing/src/harness.ts
| /** A zero'd tally for one dispatch. */ | ||
| const createDatabaseTally = (): DatabaseTally => { | ||
| return { calls: 0, durationMs: 0, errors: 0, perOperation: {}, spansEmitted: 0, spansTruncated: false }; | ||
|
|
||
| /** | ||
| * Render the running tally as span attributes. | ||
| * | ||
| * Called ONCE per dispatch, from the shard's root-span recorder — not per query. | ||
| * Building this object on every call was pure waste on a hot path. It is a handful of keys, so the per-call cost is a small object | ||
| * assignment — the property that makes `"summary"` mode scale to any call count. | ||
| */ | ||
| }; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Attach the tally documentation to formatTally.
The block after return documents neither declaration. Move it immediately above formatTally.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/do/src/database-telemetry.ts` around lines 299 - 310, Move the
documentation block currently placed after the return statement in
createDatabaseTally to immediately above the formatTally declaration, so it
documents formatTally rather than either declaration indirectly. Keep the tally
initialization and documentation text unchanged.
| const databaseAttributes = wide?.dbTally === undefined || wide.dbTally.calls === 0 ? undefined : formatTally(wide.dbTally); | ||
| const collected = | ||
| wide?.collector === undefined | ||
| ? undefined | ||
| : { ...wide.collector.collected, attributes: { ...databaseAttributes, ...wide.collector.collected.attributes } }; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Preserve database summary attributes without ctx.span.
When ctx.trace or database "spans" instrumentation causes the root-span gate to pass, but no ctx.span call created a collector, collected becomes undefined and drops the accumulated db.* attributes. Build an empty collection around databaseAttributes when a tally exists.
Proposed fix
- const collected =
- wide?.collector === undefined
- ? undefined
- : { ...wide.collector.collected, attributes: { ...databaseAttributes, ...wide.collector.collected.attributes } };
+ const collector = wide?.collector?.collected;
+ const collected =
+ collector === undefined && databaseAttributes === undefined
+ ? undefined
+ : {
+ attributes: { ...databaseAttributes, ...collector?.attributes },
+ events: collector?.events ?? [],
+ links: collector?.links ?? [],
+ };📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const databaseAttributes = wide?.dbTally === undefined || wide.dbTally.calls === 0 ? undefined : formatTally(wide.dbTally); | |
| const collected = | |
| wide?.collector === undefined | |
| ? undefined | |
| : { ...wide.collector.collected, attributes: { ...databaseAttributes, ...wide.collector.collected.attributes } }; | |
| const collector = wide?.collector?.collected; | |
| const collected = | |
| collector === undefined && databaseAttributes === undefined | |
| ? undefined | |
| : { | |
| attributes: { ...databaseAttributes, ...collector?.attributes }, | |
| events: collector?.events ?? [], | |
| links: collector?.links ?? [], | |
| }; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/do/src/shard-do.ts` around lines 5323 - 5327, Update the
collected-building logic near databaseAttributes so databaseAttributes are
preserved even when wide.collector is undefined; create an empty collection
containing the database attributes whenever a tally exists, while retaining
existing collector attributes and merge behavior when a collector is present.
A `tailSampler` that throws fails open, so every trace is kept and the sink behaves exactly like one with no sampler configured. The only symptom was an export volume nobody could explain. The failure is now reported to `console.error`, bounded twice over: once per flush window (a window with 50 traces reported 50 identical lines otherwise) and at most five times per sink instance, so a permanently broken predicate cannot become the log volume that sampling exists to hold down. The fifth line says it is going quiet. Fail-open stays, and the reasoning is now written down next to the catch rather than implied: it is deliberately the opposite of `postProcess`, which fails closed because it guards PII. A broken sampler can only mis-judge volume, and failing closed there would erase precisely the pathological traces worth keeping. The option doc now says outright that this hook is a cost control, not a guarantee. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Swept every review signal on this PR against the current code. All 7 inline threads and both outside-diff findings were already fixed and resolved; one nitpick was still genuinely open, and it's now addressed in 🔵 Make tail-sampler failures observable and boundedYou were right that this one was invisible. Now reported to
On fail-open vs. the configurable fail-closed policyKept fail-open, and wrote the reasoning down next to the catch instead of leaving it as a one-line assertion — because the PR now has the two hooks failing in opposite directions and that deserved an explanation:
I didn't add the configurable fail-closed option. A config flag would imply Two regression tests: one asserts a 3-trace window reports once (not three times), one drives 12 flush windows and asserts it stops at 5 with the silencing notice. Verification: The last CodeRabbit run was cut short by a review rate limit, so |
|
🐇🔍 ✏️ Learnings added
✅ Action performedFull review finished. Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 46 minutes. |
Two CI gates were red on this branch and neither is caught by anything run locally by default. `Lint (package.json sort)` — the `@opentelemetry/api` optional peer added for the `@lunora/server/otel` subpath left `engines` above `peerDependencies`/`peerDependenciesMeta` instead of below. Exactly the gotcha AGENTS.md documents: ESLint, Prettier, `lint:types`, `api:check` and `dist:check` are all blind to key order, so it goes green locally and red in CI. Pure key reorder, no value changes. `Lint (api surface)` — the wide-event work is a real public API addition that was never snapshotted: `SpanKind`, `SpanLink`, `SpanOptions` and `LunoraWideEvent` across server, do, runtime, testing and the lunorash umbrella re-exports. The four non-additive lines are backward-compatible widenings, consistent with this PR's no-breaking-changes claim: `ContextTracer`/`LunoraTracer` widen their third parameter from `LogFields` to `LogFields | SpanOptions`, and the protected `recordUserLog` gains two optional trailing parameters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
CI is green apart from CodSpeed, and the two lint gates that were red are fixed in
The CodSpeed reportThe two regressions I introduced are gone. The one remaining regression is not from this PR.
CodSpeed footnotes the cause itself: "No successful run was found on So this needs an acknowledgement on CodSpeed rather than a code change — I've left it rather than acknowledging it, since that's your call. Happy to re-run the benchmarks first if you'd prefer a clean comparison against a real |
Alpha had moved 56 commits and shipped its own OTel/OTLP stack (#189), which this branch had been carrying a parallel implementation of. The merge resolves that duplication in alpha's favour and re-applies the platform abstraction on top. **Alpha owns telemetry.** Its implementation is not merely the incumbent, it is better in ways this branch would have regressed: - `dispatchSpanKey(anchor)` keys wide events on `traceId:rootSpanId`, not `traceId` alone. Two concurrent RPCs forwarded under one client trace share a trace id, so keying by it merged their wide events and let whichever finished first delete the other's. - `instrumentDb` bails before touching `dispatchSpans` when instrumentation is off, and `makeDispatchSpan` leaves the map alone when there is no sink — both measured hot-path costs. - `webSocketMessage` is deliberately NOT trigger-traced: frame rate is unbounded, and minting an anchor per frame regressed fan-out to 128 members by ~30%. - The `ctx.db` tally rides `wide.dbTally`/`formatTally`, the same fold-once-per-dispatch property this branch had implemented separately. So `observability-sinks.ts`, `otlp-export.ts`, `database-telemetry.ts`, `context-telemetry.ts`, `trace-context.ts`, `create-worker.ts`, `observability.ts`, `harness.ts` and `shared/otlp.ts` all come from alpha wholesale. Dropped `@lunora/otel-bridge`: it duplicates the `createOtelTracer` bridge alpha ships as the `@lunora/server/otel` subpath, and two implementations of one bridge is worse than either. The inbound-sampled-bit fix this branch carried now lives in its own PR against alpha, so it is not duplicated here. **This branch owns the platform seam.** `shard-do.ts` could not be merged textually — it had diverged +1060/-537 here against +1277/-779 on alpha, with code moved on both sides, so a three-way merge produced 553- and 241-line "conflicts" that were really re-authoring. It is instead taken from alpha and the rebind re-applied deliberately: - 21 engine modules now resolve from `@lunora/shard-engine` - `fetch`/`alarm` delegate through `ShardRunner`, with the Cloudflare bodies injected as `handleFetchCloudflare`/`handleAlarmCloudflare` - 49 `this.state.*` reach-throughs become `shardHost.sql`, `runner.shardKey`, `runner.sockets()`, `shardHost.waitUntil` - the socket seam moves onto `ShardSocketLike`, including the boundary conversion in `webSocketMessage`/`webSocketClose` — per-socket memo state keys on the handle, so the raw socket the runtime hands back must be mapped or the two paths key on different objects Verified: 1129 `@lunora/do` tests, 115 shard-engine, 20 conformance, 8 platform-cloudflare, 741 runtime, 882 codegen, 445 server, 115 testing. The two load-bearing seams are mutation-checked — bypassing `socketFor` fails 13 tests, stubbing `shardHost.sql` fails 108. The `fetch` runner delegation is a pass-through today and fails nothing; it is structure for the next slice, and the conformance TCK is what guards the contracts. Also registers the platform family with the API-surface guard: a contracts-only package's surface IS the product, so silent drift there is exactly what that gate exists to catch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Closes the gaps between our OpenTelemetry support and
evanderkoogh/otel-cf-workers, and adds a wide-event API for instrumenting a request without drowning in log lines.Why
An audit against
otel-cf-workersturned up 11 gaps. Alpha has since closed two of them independently and better than the original plan did — resource attributes (incl. runtime detection) and inboundtraceparentadoption — so this PR builds on that work rather than duplicating it. What remains:fetchpath tracedqueue()/scheduled()/ DOalarm()/ socket frames all get a root spanctx.dbandctx.fetchctx.fetchinjectstraceparentnaming the span it recordsSPAN_KIND_INTERNALkind, plus span events, links andrecordExceptionpostProcessortailSampler— the buffer is what makes it possible@opentelemetry/apiinterop@lunora/server/otelsubpathThe subrequest problem
otlpSinkbuffered nothing — every span, log line and metric was its ownfetch. A Worker is capped at 50 (free) / 1000 (paid) subrequests per invocation, so a well-instrumented handler could spend the budget its own business logic needs. Now events buffer and ship as one request per signal, flushed at each invocation boundary viaObservabilitySink.flush.Buffering is also what makes tail sampling possible: by flush time a trace has settled, so
tailSamplercan keep it on "anything failed or ran slow", and a dropped trace takes its correlated log records with it instead of leaving orphans.A flush window is grouped by resource before wrapping — with
detectResourceson, two requests in one window can carry different host attributes, and an envelope has exactly one resource.Wide events (
ctx.span)ctx.spanis a handle on the span the request already emits. Facts accumulate across the dispatch and export as one OTel Event record (event.name: "lunora.dispatch"), so cost is flat however much you attach, and "p99 checkout latency for pro-plan users with >10 items" is one filter rather than a join across log lines.It's a log record rather than span attributes because the DO's dispatch span shares its span id with the runtime's SERVER span — exporting both would put two partial spans with identical ids on the wire.
ctx.log.event(name, fields)adds the discrete-event counterpart (LogRecord.eventNameand theevent.nameattribute, so old and new collectors both understand it).ctx.dbinstrumentation is tieredDefaulting to on required a mode that can't blow up a trace:
"summary"(default) — flat-cost counters (db.calls,db.duration_ms, per-op breakdown) folded onto the wide event. No new spans, no new records, cost independent of call count."spans"— full waterfall, capped per dispatch, reportingdb.spans_truncatedrather than silently truncating."off"— returns the db untouched (identity, not a proxy).@lunora/server/otelcreateOtelTracer(ctx)returns a real OTelTracerwhose spans arectx.tracespans, so a library instrumented against@opentelemetry/api(e.g. the AI SDK'sexperimental_telemetry.tracer) stops emitting into the Worker's no-op global provider. The two lifetime models are bridged by suspending thectx.tracebody on a promise thatend()resolves.No ambient context: tracking a current span across
awaits needsAsyncLocalStorage, which DOs avoid — so spans parent to the dispatch and the tracer is passed explicitly. Flatter than a full SDK, never wrong, never lost. Shipped as a subpath, not a package: 400 lines don't warrant their own release stream, and an optional peer dependency is how this repo already handles third-party integrations (@lunora/ai,@lunora/payment,@lunora/hyperdrive). Nothing installs@opentelemetry/apiunless it imports the subpath, and the separate packem entry keeps it out of the main@lunora/serverbundle. It's a peer rather than a dependency because the API keeps a module-global registry — a second copy would silently split it.Compatibility
No breaking changes. Batching is on by default (
batch: falserestores per-event POSTs);traceFetchandinstrumentDatabaseare opt-out;ctx.span/ctx.log.eventare additive. Alpha'sresourceAttributesbag keeps working and the loose 4-argwrapResource*call sites are unchanged.Verification
Repo-wide
lint:types,lint:eslint,lint:prettierclean;pnpm run testgreen across all 50 projects. New tests cover batching, tail sampling (including that a dropped trace takes its logs), post-processor fail-open, span kind/events/links, the traced fetch (header names the recorded span; query string stripped fromurl.full), db instrumentation tiering and its cap, wide events end-to-end via the harness, and the bridge.🤖 Generated with Claude Code
Summary by CodeRabbit
spanaccess plus span events, links, and exception recording;SpanOptions/SpanKindsupport.ctx.log.event(name, fields?)for structured, named log events.off/summary/spans) and outboundfetchtracing with optionaltraceparentpropagation.