Skip to content

feat(otel): close otel-cf-workers gaps and add wide events - #189

Merged
prisis merged 19 commits into
alphafrom
feat/otel-parity
Jul 25, 2026
Merged

prisis merged 19 commits into
alphafrom
feat/otel-parity

Conversation

@prisis

@prisis prisis commented Jul 25, 2026 •

Copy link
Copy Markdown
Member

Closes the gaps between our OpenTelemetry support and evanderkoogh/otel-cf-workers, and adds a wide-event API for instrumenting a request without drowning in log lines.

Why

An audit against otel-cf-workers turned up 11 gaps. Alpha has since closed two of them independently and better than the original plan did — resource attributes (incl. runtime detection) and inbound traceparent adoption — so this PR builds on that work rather than duplicating it. What remains:

Gap Fix
One HTTP subrequest per span/log/metric Batching exporter, one request per signal per invocation
Only the fetch path traced queue() / scheduled() / DO alarm() / socket frames all get a root span
Nothing below the dispatch span Opt-out auto-instrumentation for ctx.db and ctx.fetch
No outbound trace context ctx.fetch injects traceparent naming the span it records
Everything SPAN_KIND_INTERNAL kind, plus span events, links and recordException
No redaction seam postProcessor
Head sampling only Real tailSampler — the buffer is what makes it possible
No @opentelemetry/api interop New @lunora/server/otel subpath

The subrequest problem

otlpSink buffered nothing — every span, log line and metric was its own fetch. A Worker is capped at 50 (free) / 1000 (paid) subrequests per invocation, so a well-instrumented handler could spend the budget its own business logic needs. Now events buffer and ship as one request per signal, flushed at each invocation boundary via ObservabilitySink.flush.

Buffering is also what makes tail sampling possible: by flush time a trace has settled, so tailSampler can keep it on "anything failed or ran slow", and a dropped trace takes its correlated log records with it instead of leaving orphans.

A flush window is grouped by resource before wrapping — with detectResources on, two requests in one window can carry different host attributes, and an envelope has exactly one resource.

Wide events (ctx.span)

ctx.span.setAttributes({ "user.plan": user.plan, "cart.items": cart.length });
const payment = await charge(cart);
ctx.span.setAttributes({ "payment.provider": payment.provider });
if (payment.retried) ctx.span.addEvent("payment.retried", { attempts: payment.attempts });

ctx.span is a handle on the span the request already emits. Facts accumulate across the dispatch and export as one OTel Event record (event.name: "lunora.dispatch"), so cost is flat however much you attach, and "p99 checkout latency for pro-plan users with >10 items" is one filter rather than a join across log lines.

It's a log record rather than span attributes because the DO's dispatch span shares its span id with the runtime's SERVER span — exporting both would put two partial spans with identical ids on the wire.

ctx.log.event(name, fields) adds the discrete-event counterpart (LogRecord.eventName and the event.name attribute, so old and new collectors both understand it).

ctx.db instrumentation is tiered

Defaulting to on required a mode that can't blow up a trace:

  • "summary" (default) — flat-cost counters (db.calls, db.duration_ms, per-op breakdown) folded onto the wide event. No new spans, no new records, cost independent of call count.
  • "spans" — full waterfall, capped per dispatch, reporting db.spans_truncated rather than silently truncating.
  • "off" — returns the db untouched (identity, not a proxy).

@lunora/server/otel

createOtelTracer(ctx) returns a real OTel Tracer whose spans are ctx.trace spans, so a library instrumented against @opentelemetry/api (e.g. the AI SDK's experimental_telemetry.tracer) stops emitting into the Worker's no-op global provider. The two lifetime models are bridged by suspending the ctx.trace body on a promise that end() resolves.

No ambient context: tracking a current span across awaits needs AsyncLocalStorage, which DOs avoid — so spans parent to the dispatch and the tracer is passed explicitly. Flatter than a full SDK, never wrong, never lost. Shipped as a subpath, not a package: 400 lines don't warrant their own release stream, and an optional peer dependency is how this repo already handles third-party integrations (@lunora/ai, @lunora/payment, @lunora/hyperdrive). Nothing installs @opentelemetry/api unless it imports the subpath, and the separate packem entry keeps it out of the main @lunora/server bundle. It's a peer rather than a dependency because the API keeps a module-global registry — a second copy would silently split it.

Compatibility

No breaking changes. Batching is on by default (batch: false restores per-event POSTs); traceFetch and instrumentDatabase are opt-out; ctx.span / ctx.log.event are additive. Alpha's resourceAttributes bag keeps working and the loose 4-arg wrapResource* call sites are unchanged.

Verification

Repo-wide lint:types, lint:eslint, lint:prettier clean; pnpm run test green across all 50 projects. New tests cover batching, tail sampling (including that a dropped trace takes its logs), post-processor fail-open, span kind/events/links, the traced fetch (header names the recorded span; query string stripped from url.full), db instrumentation tiering and its cap, wide events end-to-end via the harness, and the bridge.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added richer tracing APIs: request-level span access plus span events, links, and exception recording; SpanOptions/SpanKind support.
    • Added ctx.log.event(name, fields?) for structured, named log events.
    • Added automatic database telemetry instrumentation (off/summary/spans) and outbound fetch tracing with optional traceparent propagation.
    • Improved trigger/worker observability with guaranteed telemetry flushing and wider event export; also supports traced queue/cron handling.
    • Enhanced OTLP exporting with buffered tail sampling and configurable post-processing.
    • Added testing-harness support for recording and retrieving wide dispatch span data.
  • Documentation
    • Expanded bundled third-party license details (including OpenTelemetry/Apache 2.0).

prisis and others added 6 commits July 25, 2026 09:05
Three additions the exporter needs, on top of the resource work
already on alpha.

shared/otlp-batch.ts is a minimal BatchSpanProcessor reduced to
what a Workers isolate needs: no background thread, no unbounded
queue, no retry state machine. A Worker is capped at 50/1000
subrequests per invocation, so one fetch per span is a correctness
problem, not only an efficiency one.

The wrapResource* envelopes now take one item or many, which is
what lets a whole invocation ship in a single request.

SpanEvent gains kind, events and links; SpanHandle gains addEvent,
addLink, recordException and spanContext. A trace where everything
is INTERNAL yields no service map, and a HANDLED exception
previously had nowhere to go — nothing re-throws it for a
top-level handler to log.

LogEvent gains eventName for OTel's Events API, so a structured
event is distinguishable from a human-readable log line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three gaps, one theme: make a request observable without making
the developer write a log line per fact.

ctx.span is a handle on the span the dispatch ALREADY emits.
Attributes accumulate across the request and export as one OTel
Event record (lunora.dispatch) — flat cost however much you
attach. It is a log record rather than span attributes because the
DO's root span shares its span id with the runtime's SERVER span,
so exporting both would put two partial spans with identical ids
on the wire. ctx.log.event adds the discrete-event counterpart.

ctx.fetch is instrumented: a CLIENT span plus a traceparent naming
the span it actually records. The id is minted before the request
precisely so the header cannot point at a span that never existed.

ctx.db instrumentation is tiered so it can default to on:
"summary" folds flat-cost counters onto the wide event, "spans"
gives the full waterfall capped per dispatch (reporting the
truncation rather than hiding it), "off" returns the db untouched.

alarm() and webSocketMessage() get the same trigger treatment the
worker's queue/cron entries do.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Types for the wide-event API. The doc comments carry the rule
worth internalising: log.* for narration you would read while
debugging, log.event for anything you would put on a dashboard,
and ctx.span for facts about the request itself.

ctx.trace's third argument now also accepts SpanOptions (kind,
links). It is read as options only when EVERY key is one of
attributes/kind/links, so an ordinary attribute bag is unaffected
and { attributes: { kind: ... } } resolves the one ambiguity.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Instrumentation that cannot be asserted on does not get
maintained. ctx.span in the harness now records rather than
discards, and t.wideEvent() exposes what a handler attached, so
"did we record the right thing?" is answerable in a test instead
of only after a deploy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
buildCtx now resolves the telemetry sink and trace anchor BEFORE
ctx.db, because the database is wrapped in auto-instrumentation
that needs both plus the dispatch's wide-event handle. ctx.trace
and ctx.span share one anchor so they land in the same trace.

Golden fixture regenerated; examples/expo picked up the same
change from a test run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
otlpSink buffered nothing: every span, log line and metric was its
own fetch, so a well-instrumented handler could spend the
invocation's whole subrequest budget on telemetry. It now buffers
and ships one request per signal, flushed at the invocation
boundary via the new ObservabilitySink.flush.

Buffering is also what makes real tail sampling possible: by flush
time a trace has settled, so tailSampler can keep it on "anything
failed or ran slow" — and a dropped trace takes its correlated log
records with it rather than leaving orphans.

The encoders in otlp-export move to returning bare objects so a
batch can share one envelope, and a flush window is grouped by
RESOURCE before wrapping: with detectResources on, two requests in
one window can carry different host attributes and an envelope has
exactly one resource, so mixing them would mis-attribute the batch.

Also adds postProcessor (the redaction seam, failing open so a bad
rule shows as un-redacted data rather than as silence), span
kind/events/links encoding, and LogRecord.eventName.

Builds on alpha's resource work rather than duplicating it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@netlify

netlify Bot commented Jul 25, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit 6c1e4b1
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a64c6abf518f000088a67c7
😎 Deploy Preview https://deploy-preview-189--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Jul 25, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@prisis, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 32 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8cd31989-31bd-46e6-bb43-612c5c5d1c2a

📥 Commits

Reviewing files that changed from the base of the PR and between a746e5b and 6c1e4b1.

⛔ Files ignored due to path filters (5)
  • api-snapshots/do.api.md is excluded by none and included by none
  • api-snapshots/lunora.api.md is excluded by none and included by none
  • api-snapshots/runtime.api.md is excluded by none and included by none
  • api-snapshots/server.api.md is excluded by none and included by none
  • api-snapshots/testing.api.md is excluded by none and included by none
📒 Files selected for processing (1)
  • packages/server/package.json

Walkthrough

This PR expands observability with span events, links, exceptions, database and fetch instrumentation, trigger tracing, buffered OTLP export, an OpenTelemetry tracer bridge, wide-event testing support, and bundled OpenTelemetry licensing.

Changes

Observability platform

Layer / File(s) Summary
Public tracing API and OpenTelemetry bridge
packages/server/src/types.ts, packages/server/src/otel.ts, packages/server/src/index.ts, packages/server/package.json
Adds span options, links, events, request-context span handles, and an exported OpenTelemetry tracer adapter.
Span collection and automatic instrumentation
packages/do/src/context-telemetry.ts, packages/do/src/database-telemetry.ts, packages/codegen/src/emit.ts
Collects span data, instruments outbound fetches and database operations, and wires generated contexts to dispatch spans.
Durable Object dispatch and trigger tracing
packages/do/src/shard-do.ts
Adds wide-event export, structured logger events, bounded dispatch collectors, trigger trace roots, and sink flushing for RPC, WebSocket, and alarm paths.
Worker triggers and OTLP delivery
packages/runtime/src/create-worker.ts, packages/runtime/src/observability.ts, packages/runtime/src/observability-sinks.ts, packages/runtime/src/otlp-export.ts
Instruments queue and cron handlers, adds flush hooks, buffers and samples OTLP signals, applies post-processing, encodes events and links, and sends compressed payloads.
Harness recording and bundled licensing
packages/testing/src/harness.ts, packages/*/LICENSE.md
Records shared wide events in tests and adds the bundled @opentelemetry/api Apache-2.0 license text.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Handler
  participant ShardDO
  participant ContextTelemetry
  participant OTLPBatcher
  participant OTLPExporter
  Handler->>ShardDO: invoke dispatch or trigger
  ShardDO->>ContextTelemetry: create trace and span collector
  Handler->>ContextTelemetry: record events, links, fetch, or database telemetry
  ContextTelemetry->>OTLPBatcher: buffer correlated signals
  ShardDO->>OTLPBatcher: flush at invocation boundary
  OTLPBatcher->>OTLPExporter: encode and send OTLP payloads
Loading

Possibly related PRs

Suggested labels: package: advisor, package: ai, package: db, package: queue, package: scheduler, package: studio, package: vite

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description is thorough but does not follow the required template and omits Linked issues, Test plan, Checklist, Notes, and the CLA statement. Reformat it to match the template and add a Summary, Linked issues, Test plan checklist, Checklist items, Notes for reviewers, and the exact CLA line.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change: OpenTelemetry gap-filling plus wide events.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/otel-parity

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

A library instrumented against @opentelemetry/api cannot see
Lunora's hand-rolled trace: its spans go to the global provider,
which in a Worker is the no-op one, and vanish.

createOtelTracer(ctx) returns a real OTel Tracer whose spans ARE
ctx.trace spans. The two lifetime models are bridged by suspending
the ctx.trace body on a promise that end() resolves, so Lunora
measures the real interval and no part of the runtime needs a
special case.

There is deliberately no ambient context: tracking a current span
across awaits needs AsyncLocalStorage, which DOs avoid. Spans
parent to the dispatch and the tracer is passed explicitly (the AI
SDK's experimental_telemetry.tracer) — flatter than a full SDK,
never wrong, never lost.

Shipped as the @lunora/server/otel subpath rather than its own
package: 400 lines do not warrant a release stream, and an
OPTIONAL peer dependency is how this repo already handles
third-party integrations (@lunora/ai, @lunora/payment,
@lunora/hyperdrive). Nothing installs @opentelemetry/api unless it
imports the subpath, and the separate packem entry keeps it out of
the main bundle. It is a peer rather than a dependency because the
API keeps a module-global registry, so a second copy would
silently split it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/runtime/src/observability-sinks.ts (1)

992-1032: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

combineSinks doesn't forward the new flush hook — buffered signals from a wrapped batching sink can be silently lost.

fanOut's method union ("onLog" | "onMetric" | "onRpc" | "onSpan") and the returned sink object never reference flush. Since flushSink treats a missing flush as a no-op (if (!sink?.flush) return;), any batching sink (e.g. this file's own otlpSink, which now batches by default) wrapped through combineSinks — the pattern this very module's docs recommend ("wired behind {@link combineSinks} alongside a network sink") — never receives the invocation-boundary flush call. Buffered telemetry then depends entirely on the batcher's internal maxDelayMs/maxItems timers, which can miss the isolate-freeze window the new flush hook exists specifically to guard against, per this file's own docs on ObservabilitySink.flush.

🐛 Proposed fix to forward `flush` to every child sink
     return {
+        flush: (context?: ObservabilitySinkContext) => {
+            for (const sink of sinks) {
+                if (!sink.flush) {
+                    continue;
+                }
+
+                try {
+                    sink.flush(context);
+                } catch {
+                    // Isolate failures so one bad sink doesn't starve the rest.
+                }
+            }
+        },
         onLog: (event: LogEvent, context?: ObservabilitySinkContext) => {
             fanOut("onLog", event, context);
         },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/runtime/src/observability-sinks.ts` around lines 992 - 1032, Update
combineSinks and its fanOut method to include the ObservabilitySink flush hook,
forwarding each flush invocation to every child sink that implements flush while
isolating individual failures consistently with the existing signal handlers.
Ensure the returned combined sink exposes flush so flushSink invokes wrapped
batching sinks.
🧹 Nitpick comments (2)
packages/runtime/src/create-worker.ts (1)

3898-3904: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Extract the waitUntil sink-context construction.

The same context.waitUntil ? { waitUntil: (promise) => { context.waitUntil?.(promise); } } : undefined shape now appears in instrumentTrigger and in the fetch finally. A tiny sinkContextFor(context) helper keeps the two boundaries from drifting.

Also applies to: 4316-4330

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/runtime/src/create-worker.ts` around lines 3898 - 3904, Extract the
repeated conditional sink-context construction into a shared
sinkContextFor(context) helper, preserving the existing undefined behavior when
context.waitUntil is absent and forwarding promises to context.waitUntil when
present. Replace the inline constructions in instrumentTrigger and the fetch
finally path with this helper so both boundaries use the same implementation.
packages/runtime/src/otlp-export.ts (1)

18-24: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Redundant parentSpanId assignment — spread (line 89) and the if block (lines 99-101) do the same thing.

Both set span.parentSpanId from event.parentSpanId under the same condition; the if block is now dead weight left over after the new conditional spread was added.

♻️ Proposed cleanup
     const span: Record<string, unknown> = {
         attributes,
         endTimeUnixNano: otlpUnixNano(endMs),
         // SPAN_KIND_SERVER — a dispatched RPC is server-side request handling.
         kind: OTLP_SPAN_KIND.server,
         name: event.functionPath,
         // Set only when the worker joined an upstream trace; omitted otherwise, so
         // a self-originated dispatch stays the root rather than dangling off a
         // parent that was never exported.
         ...(event.parentSpanId === undefined ? {} : { parentSpanId: event.parentSpanId }),
         // Reuse the dispatch's trace context when the runtime set it (so this span
         // shares the id it propagated as `traceparent`); else mint fresh ids.
         spanId: event.spanId ?? otlpRandomHex(8),
         startTimeUnixNano: otlpUnixNano(endMs - event.durationMs),
         // STATUS_CODE_OK (1) / STATUS_CODE_ERROR (2).
         status: event.ok ? { code: 1 } : { code: 2, message: event.error?.message ?? "" },
         traceId: event.traceId ?? otlpRandomHex(16),
     };

-    if (event.parentSpanId !== undefined) {
-        span.parentSpanId = event.parentSpanId;
-    }
-
     if (event.traceFlags !== undefined) {
         span.flags = event.traceFlags;
     }
As per path instructions, "Follow DRY principles."

Also applies to: 80-101, 119-120

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/runtime/src/otlp-export.ts` around lines 18 - 24, Remove the
redundant parentSpanId assignment from the conditional block in otlpTraceBody,
leaving the existing conditional spread as the sole assignment from
event.parentSpanId. Preserve the surrounding span construction and all other
conditional fields unchanged.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/do/src/database-telemetry.ts`:
- Around line 289-307: Change the telemetry flow around publishTally and the
dispatch-boundary handling so summary-mode database tallies do not materialize
deps.span or count as a handler-wide event on every ctx.db call. Publish the
accumulated tally once at the dispatch boundary, or route summary attributes
through a non-wide-event channel, while preserving spans-mode emission and
ensuring shard-do.ts’s root-span gate only fires when the handler actually
records a wide event.

In `@packages/do/src/shard-do.ts`:
- Around line 205-215: Reposition the detached JSDoc blocks in
packages/do/src/shard-do.ts: move the block currently spanning lines 192–204
immediately above fuseCloudflareTraces, move the 4641–4658 block immediately
above makeDispatchSpan, and move the 4677–4689 block immediately above
makeFetch. Keep each declaration’s existing behavior unchanged and ensure no
documentation block remains separated from the member it describes.
- Around line 1950-1967: Update the dispatchSpans keying from traceId-only to
the anchor span identity, such as a traceId/rootSpanId combination, so
concurrent dispatches sharing a trace use separate wide-event collectors. Apply
the same key derivation consistently in makeDispatchSpan, the fetch finally
cleanup, withTriggerTrace, and recordDispatchRootSpan, preserving per-dispatch
cleanup and root-span gating.

In `@packages/runtime/src/create-worker.ts`:
- Around line 3906-3915: Update the trigger-event emissions in the run wrapper
around emitRpcEvent so they are exported independently of head-trace sampling,
or use a dedicated trigger sampling ratio. Ensure both success and error events
remain visible for low-frequency triggers while preserving their existing event
payloads and error propagation.

In `@packages/server/src/otel.ts`:
- Around line 242-246: Update Span.recordException to normalize non-Error
Exception objects before passing them to handle.recordException, preserving
string and Error inputs while converting object values to a message-bearing
exception using their message and available fields.

In `@packages/testing/src/harness.ts`:
- Around line 334-339: Update the recordException callback in SpanHandle to
include the documented exception.stacktrace field when recording the "exception"
event. Preserve the existing Error checks and type/message behavior, using the
Error stack when available and an appropriate fallback for non-Error values so
harness events match production instrumentation.

---

Outside diff comments:
In `@packages/runtime/src/observability-sinks.ts`:
- Around line 992-1032: Update combineSinks and its fanOut method to include the
ObservabilitySink flush hook, forwarding each flush invocation to every child
sink that implements flush while isolating individual failures consistently with
the existing signal handlers. Ensure the returned combined sink exposes flush so
flushSink invokes wrapped batching sinks.

---

Nitpick comments:
In `@packages/runtime/src/create-worker.ts`:
- Around line 3898-3904: Extract the repeated conditional sink-context
construction into a shared sinkContextFor(context) helper, preserving the
existing undefined behavior when context.waitUntil is absent and forwarding
promises to context.waitUntil when present. Replace the inline constructions in
instrumentTrigger and the fetch finally path with this helper so both boundaries
use the same implementation.

In `@packages/runtime/src/otlp-export.ts`:
- Around line 18-24: Remove the redundant parentSpanId assignment from the
conditional block in otlpTraceBody, leaving the existing conditional spread as
the sole assignment from event.parentSpanId. Preserve the surrounding span
construction and all other conditional fields unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 53099413-a6aa-4942-85eb-e8c5121f1715

📥 Commits

Reviewing files that changed from the base of the PR and between 950f957 and 62df942.

⛔ Files ignored due to path filters (16)
  • examples/expo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • packages/codegen/__tests__/fixtures/simple/expected/_generated/shard.ts is excluded by !**/_generated/**, !**/__tests__/** and included by packages/**
  • packages/do/__tests__/database-telemetry.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/do/__tests__/traced-fetch.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/runtime/__tests__/observability-sinks.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/server/__tests__/context.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/server/__tests__/functions-v-from.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/server/__tests__/functions.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/server/__tests__/otel.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/testing/__tests__/wide-event.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml and included by none
  • pnpm-workspace.yaml is excluded by none and included by none
  • shared/log-event.ts is excluded by none and included by none
  • shared/otlp-batch.ts is excluded by none and included by none
  • shared/otlp.ts is excluded by none and included by none
  • shared/span-event.ts is excluded by none and included by none
📒 Files selected for processing (16)
  • packages/cloudflare-access/LICENSE.md
  • packages/codegen/src/emit.ts
  • packages/do/src/context-telemetry.ts
  • packages/do/src/database-telemetry.ts
  • packages/do/src/shard-do.ts
  • packages/replica/LICENSE.md
  • packages/runtime/src/create-worker.ts
  • packages/runtime/src/observability-sinks.ts
  • packages/runtime/src/observability.ts
  • packages/runtime/src/otlp-export.ts
  • packages/server/LICENSE.md
  • packages/server/package.json
  • packages/server/src/index.ts
  • packages/server/src/otel.ts
  • packages/server/src/types.ts
  • packages/testing/src/harness.ts

Comment thread packages/do/src/database-telemetry.ts Resolved
Comment thread packages/do/src/shard-do.ts Resolved
Comment thread packages/do/src/shard-do.ts Resolved
Comment thread packages/runtime/src/create-worker.ts Resolved
Comment thread packages/server/src/otel.ts Resolved
Comment thread packages/testing/src/harness.ts Resolved
@codspeed

codspeed Bot commented Jul 25, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 18.54%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 regressed benchmark
✅ 238 untouched benchmarks
⏩ 2 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ deep mixed stack 618.7 µs 759.5 µs -18.54%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing feat/otel-parity (6c1e4b1) with alpha (49b10ad)2

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on alpha (aa7abe6) during the generation of this report, so 49b10ad was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

prisis and others added 2 commits July 25, 2026 13:32
combineSinks did not forward the new flush hook, so any batching
sink wrapped through it — the pairing this module's own docs
recommend — never received the invocation-boundary flush, and its
buffer died with the isolate. fanOut now takes an argument list,
because flush(context) puts the context first while the four
on*(event, context) hooks put it second. Covered by a test.

dispatchSpans was keyed by traceId alone. resolveTraceAnchor takes
traceId from the inbound traceparent, so two concurrent RPCs under
one client trace shared an entry: their wide events merged and
whichever finished first deleted the other's. Keyed by trace id AND
root span id now. traceSampling keeping traceId is still right —
a sampling verdict is per-trace, a wide event is per-dispatch.

Summary-mode db counters wrote through ctx.span, which materializes
the wide-event collector. Since summary is the default, nearly every
request then looked like the handler had recorded a wide event,
firing the root-span gate and exporting a lunora.dispatch record for
handlers that instrumented nothing — the opposite of what the gate
exists for. The tally now lands on a separate slot that enriches a
root span being recorded anyway, but never causes one.

Trigger events no longer ride the head-sampling ratio: a ratio tuned
for request traffic hid most fires of an hourly cron, which is the
blind spot the wrapper was added to close.

Also: OTel Exception's object form keeps its message instead of
stringifying to [object Object]; dropped a dead `ok` write CodeQL
flagged (it was exactly `error === undefined`); removed a duplicated
parentSpanId assignment; extracted sinkContextFor so both invocation
boundaries share one shape; reattached four JSDoc blocks that member
reordering had orphaned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/runtime/src/observability-sinks.ts (1)

42-160: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Do not export the unprocessed signal when redaction fails.

The fail-open postProcess path can send unredacted telemetry after a redaction hook throws. This is especially risky because ObservabilityEvent.error.message may contain user input, and third-party sinks are expected to scrub it. Drop the affected signal or make unredacted fallback an explicit opt-in with a diagnostic metric.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/runtime/src/observability-sinks.ts` around lines 42 - 160, Update
postProcess and encodeSignal so a post-processor exception never returns or
encodes the original unredacted event. Drop the affected signal by returning
undefined, preserving normal hook-returned events and explicit hook drops; do
not add an unredacted fallback path.
🧹 Nitpick comments (1)
packages/runtime/src/observability-sinks.ts (1)

42-160: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

Make tail-sampler failures observable and bounded.

When tailSampler throws, the implementation keeps the entire trace, silently bypassing the configured sampling policy. Repeated failures can substantially increase export volume and cost. Emit a rate-limited diagnostic/metric and document whether fail-open is intentional; consider a configurable fail-closed policy when sampling is a volume-control guarantee.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/runtime/src/observability-sinks.ts` around lines 42 - 160, Update
the catch block in applyTailSampler to emit a rate-limited diagnostic or metric
when tailSampler throws, preventing repeated failures from generating unbounded
telemetry. Document that the current behavior intentionally fails open and
retains the trace; if the existing configuration supports sampling as a
volume-control guarantee, expose and honor a configurable fail-closed policy.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@packages/runtime/src/observability-sinks.ts`:
- Around line 42-160: Update postProcess and encodeSignal so a post-processor
exception never returns or encodes the original unredacted event. Drop the
affected signal by returning undefined, preserving normal hook-returned events
and explicit hook drops; do not add an unredacted fallback path.

---

Nitpick comments:
In `@packages/runtime/src/observability-sinks.ts`:
- Around line 42-160: Update the catch block in applyTailSampler to emit a
rate-limited diagnostic or metric when tailSampler throws, preventing repeated
failures from generating unbounded telemetry. Document that the current behavior
intentionally fails open and retains the trace; if the existing configuration
supports sampling as a volume-control guarantee, expose and honor a configurable
fail-closed policy.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 29375b51-b502-4ef9-8e06-0c8372e937b9

📥 Commits

Reviewing files that changed from the base of the PR and between 62df942 and 611c168.

⛔ Files ignored due to path filters (5)
  • examples/expo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • packages/codegen/__tests__/fixtures/simple/expected/_generated/shard.ts is excluded by !**/_generated/**, !**/__tests__/** and included by packages/**
  • packages/do/__tests__/database-telemetry.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/runtime/__tests__/observability-sinks.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml and included by none
📒 Files selected for processing (9)
  • packages/codegen/src/emit.ts
  • packages/do/src/context-telemetry.ts
  • packages/do/src/database-telemetry.ts
  • packages/do/src/shard-do.ts
  • packages/runtime/src/create-worker.ts
  • packages/runtime/src/observability-sinks.ts
  • packages/runtime/src/otlp-export.ts
  • packages/server/package.json
  • packages/server/src/otel.ts
💤 Files with no reviewable changes (1)
  • packages/runtime/src/otlp-export.ts
🚧 Files skipped from review as they are similar to previous changes (7)
  • packages/server/package.json
  • packages/runtime/src/create-worker.ts
  • packages/codegen/src/emit.ts
  • packages/do/src/database-telemetry.ts
  • packages/do/src/shard-do.ts
  • packages/server/src/otel.ts
  • packages/do/src/context-telemetry.ts

SpanHandle.recordException promises exception.type, exception.message
AND exception.stacktrace. The harness recorder omitted the last one,
so a test asserting on a wide event's exception attributes passed
against a narrower record than the one production actually ships —
which defeats the point of making the instrumentation assertable.

The remaining divergence is documented inline: production derives
exception.type via toErrorType, preferring a LunoraError's stable
code, and that helper is not exported from @lunora/do. For a plain
Error — the common case in a test — the two agree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@prisis

prisis commented Jul 25, 2026

Copy link
Copy Markdown
Member Author

Addressed all review findings. Summary of what changed and why:

🔴 combineSinks didn't forward flush — correct and the most important one. This was a regression introduced when I ported the work onto alpha: the original branch had the flush fan-out, and re-authoring otlpSink onto alpha's otlp-export.ts structure lost it. Any batching sink wrapped through combineSinks — the pairing this module's own docs recommend — never received the invocation-boundary flush, so its buffer died with the isolate. fanOut now takes an argument list, because flush(context) puts the context first while the four on*(event, context) hooks put it second; a fixed shape would have handed flush an undefined context. Added two regression tests so it can't silently come back.

🟠 dispatchSpans keyed by traceId alone — correct, and a genuine data-loss bug. resolveTraceAnchor takes traceId from the inbound traceparent, so two concurrent RPCs under one client trace shared an entry: wide events merged and whichever dispatch finished first deleted the other's. Now keyed by ${traceId}:${rootSpanId}. Your framing of why traceSampling is fine keyed by traceId — a sampling verdict is per-trace, a wide event is per-dispatch — is exactly right and is now the doc comment on the key helper.

**🟠 "summary"" mode made every db-touching dispatch look like a wide event** — correct, and it defeated the root-span gate I'd documented as deliberate. publishTallywrote throughctx.span, which materialises the collector, so with "summary"as the default nearly every request fired the gate and exported alunora.dispatch` record for handlers that instrumented nothing. Took your first suggestion: the tally now lands on a separate slot on the dispatch entry that enriches a root span being recorded anyway, but never causes one. An uninstrumented handler emits nothing extra again.

🟡 Head sampling silenced low-frequency triggers — correct. Trigger events no longer ride the ratio at all; volume is inherently low and a ratio tuned for request traffic hid most fires of an hourly cron, which is the blind spot the wrapper exists to close.

🟡 OTel Exception object form lost its message — correct, applied your normalisation.

🟡 Detached JSDoc blocks — correct, all four reattached (member reordering for member-ordering had orphaned them). Also corrected two doc comments that had gone stale against the tally change.

CodeQL: dead ok write — correct. Declaring it boolean | undefined traded the dead write for a TS "used before assigned" error, so I removed the variable instead: it was exactly error === undefined on every path, since a non-2xx response sets error just as a throw does.

Nitpicks — both applied: extracted sinkContextFor so the two invocation boundaries share one shape, and removed the duplicated parentSpanId assignment in otlp-export.ts.

One more found while reviewing your harness comment, which was also right: the harness recordException omitted exception.stacktrace, so wide-event tests passed against a narrower record than production ships. Fixed with an assertion covering it. The remaining divergence is documented inline — production prefers a LunoraError's stable code via toErrorType, which isn't exported from @lunora/do; for a plain Error the two agree.

Verification: repo-wide lint:types, lint:eslint, lint:prettier clean; pnpm run test green. Two studio logs-panel tests flake under the loaded parallel run — packages/studio is byte-identical to alpha on this branch and they pass 29/29 in three consecutive isolated runs, so that's pre-existing timing flake in waitFor + virtualization assertions, not a regression here.

prisis and others added 2 commits July 25, 2026 14:16
Redaction now fails CLOSED. postProcess previously returned the
event unmodified when a hook threw, on the reasoning that visible
un-redacted data beats silent absence. That is right for
observability in general and wrong for this hook: postProcessor
exists to strip PII before telemetry leaves for a third-party
collector, and error.message routinely carries user input. A
dropped span is a monitoring gap you notice; a leaked payload is an
incident you cannot retract. The test that asserted the old
behaviour is inverted.

Two benchmark regressions CodSpeed caught, both self-inflicted:

broadcastWhisper to 128 members (-31%) — webSocketMessage was
wrapped in withTriggerTrace, minting a trace anchor per frame (two
crypto draws plus hex encoding) on an unbounded-frequency path. The
wrapper is removed there. The trade was poor anyway: a frame's work
is a subscription re-evaluation already attributable to the RPC that
wrote the data. alarm keeps the wrapper — low frequency, and
genuinely un-attributable background work, which was the blind spot
worth closing.

rankPage hydration (-17%) — two causes. The db tally was published
per call, rebuilding a fields object and doing a map lookup on every
query; the shard now reads a plain counter object once at the
dispatch boundary, so a query pays only integer increments. And both
makeDispatchSpan and the tally touched dispatchSpans eagerly on
every buildCtx even with no sink configured; both now bail when
nothing is collecting, so a deployment that exports no telemetry
pays nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@prisis

prisis commented Jul 25, 2026

Copy link
Copy Markdown
Member Author

Correcting my earlier claim: I had not addressed everything. The 7 inline threads were fixed and are resolved, but I missed two later signals. Both are now fixed in 9f8e8ade.

🟠 Security: redaction failed open

The third review flagged that postProcess returned the event unmodified when a hook threw. I'd written that fail-open deliberately and defended the reasoning — "a redaction bug should show as un-redacted data rather than as silence." That's right for observability in general and wrong for this hook specifically: postProcessor exists to strip PII before telemetry leaves for a third-party collector, and error.message routinely carries user input. A dropped span is a monitoring gap you notice; a leaked payload is an incident you can't retract. It now fails closed, and the test that asserted the old behaviour is inverted rather than deleted.

❌ Two CodSpeed regressions, both self-inflicted

I hadn't looked at the performance report at all. Both regressions were mine:

broadcastWhisper to 128 members −31% — I'd wrapped webSocketMessage in withTriggerTrace, which mints a trace anchor per frame (two crypto draws plus hex encoding) on a path with unbounded frequency. Removed there. The trade was poor regardless: a frame's work is a subscription re-evaluation already attributable to the RPC that wrote the data. alarm keeps the wrapper — low frequency, genuinely un-attributable background work, which was the blind spot actually worth closing.

rankPage hydration −17% — two causes. The db tally was published per call, rebuilding a LogFields object and doing a map lookup on every query; the shard now reads a plain counter object once at the dispatch boundary, so a query pays only integer increments. This is closer to what the original review suggested ("publish the tally once at the dispatch boundary") than my first attempt was. Separately, both makeDispatchSpan and the tally touched dispatchSpans eagerly on every buildCtx even with no sink configured — so a deployment exporting no telemetry was paying for it. Both now bail when nothing is collecting.

Please re-run CodSpeed to confirm; local wall-clock isn't comparable to its simulated instruction counts, so I'm not claiming the numbers, only that the per-call and per-frame work is gone.

Verification: repo-wide lint:types, lint:eslint, lint:prettier clean; do 1223, runtime 708, server 430, codegen 864, testing 90 all passing after merging the latest alpha.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/do/src/database-telemetry.ts`:
- Around line 299-310: Move the documentation block currently placed after the
return statement in createDatabaseTally to immediately above the formatTally
declaration, so it documents formatTally rather than either declaration
indirectly. Keep the tally initialization and documentation text unchanged.

In `@packages/do/src/shard-do.ts`:
- Around line 5323-5327: Update the collected-building logic near
databaseAttributes so databaseAttributes are preserved even when wide.collector
is undefined; create an empty collection containing the database attributes
whenever a tally exists, while retaining existing collector attributes and merge
behavior when a collector is present.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4052e383-4c61-4838-b6e8-6a7245d46cac

📥 Commits

Reviewing files that changed from the base of the PR and between 611c168 and 4e6914c.

⛔ Files ignored due to path filters (4)
  • packages/do/__tests__/database-telemetry.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/runtime/__tests__/observability-sinks.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/testing/__tests__/wide-event.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml and included by none
📒 Files selected for processing (7)
  • packages/codegen/src/emit.ts
  • packages/do/src/database-telemetry.ts
  • packages/do/src/shard-do.ts
  • packages/runtime/src/observability-sinks.ts
  • packages/server/src/index.ts
  • packages/server/src/types.ts
  • packages/testing/src/harness.ts
🚧 Files skipped from review as they are similar to previous changes (4)
  • packages/server/src/index.ts
  • packages/codegen/src/emit.ts
  • packages/server/src/types.ts
  • packages/testing/src/harness.ts

Comment on lines +299 to +310
/** A zero'd tally for one dispatch. */
const createDatabaseTally = (): DatabaseTally => {
return { calls: 0, durationMs: 0, errors: 0, perOperation: {}, spansEmitted: 0, spansTruncated: false };

/**
* Render the running tally as span attributes.
*
* Called ONCE per dispatch, from the shard's root-span recorder — not per query.
* Building this object on every call was pure waste on a hot path. It is a handful of keys, so the per-call cost is a small object
* assignment — the property that makes `"summary"` mode scale to any call count.
*/
};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Attach the tally documentation to formatTally.

The block after return documents neither declaration. Move it immediately above formatTally.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/do/src/database-telemetry.ts` around lines 299 - 310, Move the
documentation block currently placed after the return statement in
createDatabaseTally to immediately above the formatTally declaration, so it
documents formatTally rather than either declaration indirectly. Keep the tally
initialization and documentation text unchanged.

Comment on lines +5323 to +5327
const databaseAttributes = wide?.dbTally === undefined || wide.dbTally.calls === 0 ? undefined : formatTally(wide.dbTally);
const collected =
wide?.collector === undefined
? undefined
: { ...wide.collector.collected, attributes: { ...databaseAttributes, ...wide.collector.collected.attributes } };

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Preserve database summary attributes without ctx.span.

When ctx.trace or database "spans" instrumentation causes the root-span gate to pass, but no ctx.span call created a collector, collected becomes undefined and drops the accumulated db.* attributes. Build an empty collection around databaseAttributes when a tally exists.

Proposed fix
-        const collected =
-            wide?.collector === undefined
-                ? undefined
-                : { ...wide.collector.collected, attributes: { ...databaseAttributes, ...wide.collector.collected.attributes } };
+        const collector = wide?.collector?.collected;
+        const collected =
+            collector === undefined && databaseAttributes === undefined
+                ? undefined
+                : {
+                      attributes: { ...databaseAttributes, ...collector?.attributes },
+                      events: collector?.events ?? [],
+                      links: collector?.links ?? [],
+                  };
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const databaseAttributes = wide?.dbTally === undefined || wide.dbTally.calls === 0 ? undefined : formatTally(wide.dbTally);
const collected =
wide?.collector === undefined
? undefined
: { ...wide.collector.collected, attributes: { ...databaseAttributes, ...wide.collector.collected.attributes } };
const collector = wide?.collector?.collected;
const collected =
collector === undefined && databaseAttributes === undefined
? undefined
: {
attributes: { ...databaseAttributes, ...collector?.attributes },
events: collector?.events ?? [],
links: collector?.links ?? [],
};
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/do/src/shard-do.ts` around lines 5323 - 5327, Update the
collected-building logic near databaseAttributes so databaseAttributes are
preserved even when wide.collector is undefined; create an empty collection
containing the database attributes whenever a tally exists, while retaining
existing collector attributes and merge behavior when a collector is present.

A `tailSampler` that throws fails open, so every trace is kept and the
sink behaves exactly like one with no sampler configured. The only
symptom was an export volume nobody could explain.

The failure is now reported to `console.error`, bounded twice over: once
per flush window (a window with 50 traces reported 50 identical lines
otherwise) and at most five times per sink instance, so a permanently
broken predicate cannot become the log volume that sampling exists to
hold down. The fifth line says it is going quiet.

Fail-open stays, and the reasoning is now written down next to the catch
rather than implied: it is deliberately the opposite of `postProcess`,
which fails closed because it guards PII. A broken sampler can only
mis-judge volume, and failing closed there would erase precisely the
pathological traces worth keeping. The option doc now says outright that
this hook is a cost control, not a guarantee.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@prisis

prisis commented Jul 25, 2026

Copy link
Copy Markdown
Member Author

Swept every review signal on this PR against the current code. All 7 inline threads and both outside-diff findings were already fixed and resolved; one nitpick was still genuinely open, and it's now addressed in a746e5be.

🔵 Make tail-sampler failures observable and bounded

You were right that this one was invisible. applyTailSampler caught a throwing sampler, set verdict = true, and said nothing — so a broken predicate and a permissive one are indistinguishable at runtime: every trace is kept either way, and the only symptom is a collector bill nobody can account for.

Now reported to console.error, bounded on both axes:

  • Once per flush window. The catch is per trace, so a window holding 50 traces would have emitted 50 identical lines. Failures are accumulated and reported once, carrying the first error and the affected trace count.
  • Five times per sink instance. A sampler that throws throws on every window, so an unbounded diagnostic becomes exactly the log volume tail sampling exists to hold down — on a hot Worker it would outweigh the telemetry it's complaining about. The fifth line says it's going quiet. The counter lives on the sink rather than the module, so a fresh isolate reports again and a persistently broken sampler stays visible over time instead of being silenced permanently by the first burst.

On fail-open vs. the configurable fail-closed policy

Kept fail-open, and wrote the reasoning down next to the catch instead of leaving it as a one-line assertion — because the PR now has the two hooks failing in opposite directions and that deserved an explanation:

postProcess fails closed because it guards PII, and a leaked payload can't be retracted. tailSampler fails open because it can only mis-judge volume, and dropping the trace deletes the evidence of whatever the sampler choked on. There's a sharper argument against fail-closed here too: a sampler that throws on the pathological traces — a cyclic attribute, a huge error message — would erase precisely the traces most worth keeping.

I didn't add the configurable fail-closed option. A config flag would imply tailSampler can be a volume guarantee, and it can't be: it's user code in the request path, and the only enforcement that survives a bug in it lives at the collector. The option's doc now says that outright rather than leaving a reader to infer a bound the hook can't offer.

Two regression tests: one asserts a 3-trace window reports once (not three times), one drives 12 flush windows and asserts it stops at 5 with the silencing notice.

Verification: @lunora/runtime 741 tests passing, lint:types / lint:eslint / lint:prettier clean. Extracted groupByTrace out of applyTailSampler along the way — the added branch pushed it past the sonarjs/cognitive-complexity ceiling.

The last CodeRabbit run was cut short by a review rate limit, so 9f8e8ade..a746e5be hasn't been looked at yet — @coderabbitai full review when you have budget.

@coderabbitai

coderabbitai Bot commented Jul 25, 2026 •

Copy link
Copy Markdown

@prisis Thanks for the thorough follow-up. The bounded diagnostics and the distinction between fail-closed PII redaction versus fail-open sampling are well reasoned. I’ll re-review the full PR, including 9f8e8ade..a746e5be.

🐇🔍


✏️ Learnings added
Learnt from: prisis
URL: https://github.com/anolilab/lunora/pull/189

Timestamp: 2026-07-25T14:09:04.977Z
Learning: In PR `#189`'s OpenTelemetry runtime sinks, `postProcessor` must fail closed because it may redact PII before third-party export, while `tailSampler` intentionally fails open because a sampler failure affects volume rather than data safety and dropping the failing trace can erase valuable diagnostic evidence. Tail-sampler failures are reported at most once per flush window and five times per sink instance.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

✅ Action performed

Full review finished.


Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 46 minutes.

Two CI gates were red on this branch and neither is caught by anything
run locally by default.

`Lint (package.json sort)` — the `@opentelemetry/api` optional peer added
for the `@lunora/server/otel` subpath left `engines` above
`peerDependencies`/`peerDependenciesMeta` instead of below. Exactly the
gotcha AGENTS.md documents: ESLint, Prettier, `lint:types`, `api:check`
and `dist:check` are all blind to key order, so it goes green locally and
red in CI. Pure key reorder, no value changes.

`Lint (api surface)` — the wide-event work is a real public API addition
that was never snapshotted: `SpanKind`, `SpanLink`, `SpanOptions` and
`LunoraWideEvent` across server, do, runtime, testing and the lunorash
umbrella re-exports.

The four non-additive lines are backward-compatible widenings, consistent
with this PR's no-breaking-changes claim: `ContextTracer`/`LunoraTracer`
widen their third parameter from `LogFields` to `LogFields | SpanOptions`,
and the protected `recordUserLog` gains two optional trailing parameters.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@prisis
prisis merged commit 49b10ad into alpha Jul 25, 2026
8 of 9 checks passed
@prisis
prisis deleted the feat/otel-parity branch July 25, 2026 14:22
@prisis

prisis commented Jul 25, 2026

Copy link
Copy Markdown
Member Author

CI is green apart from CodSpeed, and the two lint gates that were red are fixed in 9061c0dd:

  • Lint (package.json sort) — the @opentelemetry/api optional peer left engines above peerDependencies/peerDependenciesMeta in packages/server/package.json. Exactly the trap AGENTS.md documents: that job is the only thing in CI that sees key order, so it went green locally and red here. Pure key reorder, no value changes.
  • Lint (api surface) — the wide-event work is a real public API addition that was never snapshotted (SpanKind, SpanLink, SpanOptions, LunoraWideEvent, across server/do/runtime/testing and the lunorash re-exports). Refreshed via api:update. I checked the diff for breakage rather than assuming: of 194 lines, only four are non-additive, and all four are backward-compatible widenings — ContextTracer/LunoraTracer widen their third parameter from LogFields to LogFields | SpanOptions, and the protected recordUserLog gains two optional trailing parameters. The no-breaking-changes claim holds.

The CodSpeed report

The two regressions I introduced are gone. broadcastWhisper to 128 members (−31%) and rankPage hydration (−17%) are both back in the untouched set, so removing withTriggerTrace from webSocketMessage and moving the db tally to the dispatch boundary did what they were meant to.

The one remaining regression is not from this PR. deep mixed stack (−18.54%) lives in packages/fingerprint/__bench__/fingerprint-stack.bench.ts, and the benchmarked code is untouched:

  • This PR changes zero files under packages/fingerprint/. git diff origin/alpha HEAD -- packages/fingerprint/ is empty — the package is byte-identical to alpha.
  • @lunora/fingerprint is genuinely zero-dep, as documented. Its only imports are ./sha256, ./superlog and ./lunora. It reaches nothing in shared/, so the four shared/* files this PR does touch (otlp.ts, otlp-batch.ts, log-event.ts, span-event.ts) cannot be in its path.

CodSpeed footnotes the cause itself: "No successful run was found on alpha (aa7abe6) during the generation of this report, so 49b10ad was used instead as the comparison base. There might be some changes unrelated to this pull request in this report" — alongside the "Different runtime environments detected" warning. And packages/fingerprint is also unchanged between 49b10ad and real alpha, so the substitute base doesn't explain it as a genuine delta either. Identical code measured across different runners is the whole story.

So this needs an acknowledgement on CodSpeed rather than a code change — I've left it rather than acknowledging it, since that's your call. Happy to re-run the benchmarks first if you'd prefer a clean comparison against a real alpha baseline before dismissing it.

prisis added a commit that referenced this pull request Jul 25, 2026
Alpha had moved 56 commits and shipped its own OTel/OTLP stack (#189),
which this branch had been carrying a parallel implementation of. The
merge resolves that duplication in alpha's favour and re-applies the
platform abstraction on top.

**Alpha owns telemetry.** Its implementation is not merely the incumbent,
it is better in ways this branch would have regressed:

- `dispatchSpanKey(anchor)` keys wide events on `traceId:rootSpanId`, not
  `traceId` alone. Two concurrent RPCs forwarded under one client trace
  share a trace id, so keying by it merged their wide events and let
  whichever finished first delete the other's.
- `instrumentDb` bails before touching `dispatchSpans` when instrumentation
  is off, and `makeDispatchSpan` leaves the map alone when there is no
  sink — both measured hot-path costs.
- `webSocketMessage` is deliberately NOT trigger-traced: frame rate is
  unbounded, and minting an anchor per frame regressed fan-out to 128
  members by ~30%.
- The `ctx.db` tally rides `wide.dbTally`/`formatTally`, the same
  fold-once-per-dispatch property this branch had implemented separately.

So `observability-sinks.ts`, `otlp-export.ts`, `database-telemetry.ts`,
`context-telemetry.ts`, `trace-context.ts`, `create-worker.ts`,
`observability.ts`, `harness.ts` and `shared/otlp.ts` all come from alpha
wholesale. Dropped `@lunora/otel-bridge`: it duplicates the
`createOtelTracer` bridge alpha ships as the `@lunora/server/otel`
subpath, and two implementations of one bridge is worse than either.

The inbound-sampled-bit fix this branch carried now lives in its own PR
against alpha, so it is not duplicated here.

**This branch owns the platform seam.** `shard-do.ts` could not be merged
textually — it had diverged +1060/-537 here against +1277/-779 on alpha,
with code moved on both sides, so a three-way merge produced 553- and
241-line "conflicts" that were really re-authoring. It is instead taken
from alpha and the rebind re-applied deliberately:

- 21 engine modules now resolve from `@lunora/shard-engine`
- `fetch`/`alarm` delegate through `ShardRunner`, with the Cloudflare
  bodies injected as `handleFetchCloudflare`/`handleAlarmCloudflare`
- 49 `this.state.*` reach-throughs become `shardHost.sql`,
  `runner.shardKey`, `runner.sockets()`, `shardHost.waitUntil`
- the socket seam moves onto `ShardSocketLike`, including the boundary
  conversion in `webSocketMessage`/`webSocketClose` — per-socket memo
  state keys on the handle, so the raw socket the runtime hands back must
  be mapped or the two paths key on different objects

Verified: 1129 `@lunora/do` tests, 115 shard-engine, 20 conformance, 8
platform-cloudflare, 741 runtime, 882 codegen, 445 server, 115 testing.
The two load-bearing seams are mutation-checked — bypassing `socketFor`
fails 13 tests, stubbing `shardHost.sql` fails 108. The `fetch` runner
delegation is a pass-through today and fails nothing; it is structure for
the next slice, and the conformance TCK is what guards the contracts.

Also registers the platform family with the API-surface guard: a
contracts-only package's surface IS the product, so silent drift there is
exactly what that gate exists to catch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — 6c1e4b15 Deployed Jul 25, 2026 by prisis via Benchmarks #959
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant