Skip to content

feat(cloud): full OpenTelemetry integration (generations, protobuf, hardened ingest, tiering) - #161

Merged
prisis merged 9 commits into
feat/cloud-traces-viewerfrom
feat/cloud-otlp-full
Jul 22, 2026
Merged

prisis merged 9 commits into
feat/cloud-traces-viewerfrom
feat/cloud-otlp-full

Conversation

@prisis

@prisis prisis commented Jul 21, 2026

Copy link
Copy Markdown
Member

What

Completes the full-OpenTelemetry integration on the cloud side, stacked on the Traces work (#157). Pairs with framework #160 (AI generation-span emission → alpha). All 9 planned phases landed.

Phase Commit Summary
5 bc335622 Generation depth — observations gains kind:"generation" (model, prompt/completion tokens, opt-in input/output); decoder promotes any gen_ai.request.model span; waterfall shows a gen chip
6 d67d6000 Filters + fix — traces.list gains errorOnly/minDurationMs/functionPath/from/to; a by_org_deployment_started index fixes quiet deployments' older traces vanishing past the 2000-span window
4 ec918166 Full OTLP — Worker-safe hand-rolled protobuf decoder (protobufjs needs eval, blocked in Workers), /v1/metrics → Analytics Engine, partialSuccess on cap
3 ae89b149 Hardened ingest — scoped ingest deploy-key capability (can't deploy), per-org telemetry rate-limit tier, key-based PII redaction
2 e3b7361d Deploy wiring — provisioner injects LUNORA_OTLP_ENDPOINT/token + tail_consumers; per-org ingest key stored envelope-encrypted for re-injection
8 b0c9f1c2 Detail pane — click a waterfall span → model/tokens/io/attributes
7 21b8752a Tiering — archiveSpans streams spans to Pipeline → R2/Iceberg (scales past D1)
9 540dfea6 Tests + GAPS — OTLP route handler coverage (auth, protobuf-accepted, partialSuccess)

Why

The Traces viewer (#157) had a full span store but the value was bottlenecked and the ingest incomplete. This makes the cloud a real, hardened OpenTelemetry endpoint: any SDK/Collector ships over JSON or protobuf, safely (scoped key, rate-limit, redaction); AI generations show cost/tokens; traces filter/paginate and tier to columnar storage; and telemetry flows automatically once a tenant is deployed.

Key discovery: the framework's deep-span stack (ctx.trace, ctx.metrics) was already on alpha — so /v1/metrics ingest became load-bearing and generations became the framework's only missing emit (shipped in #160).

Not included (🌐 / follow-on, tracked in GAPS.md B2)

  • R2-SQL read-back of archived spans (an action over the Iceberg table) — the write side lands here, like the existing raw-event archive.
  • Live end-to-end run against a dispatch namespace with LUNORA_OTLP_ENDPOINT set.
  • Cross-tab UI links (trace→logs, error-span→Issue) — need dashboard tab-state + the framework carrying traceId into getIssues.
  • Phase 0 rebase onto alpha for the eslint TS-6 pin (deferred — branches diverged; do deliberately).

Test plan

  • 315 cloud tests pass (pnpm --filter @lunora/cloud run test), tsc --noEmit clean, codegen regenerated.
  • New coverage: generation decode, trace filters, protobuf round-trip, metrics decode, PII redaction, WfP tail_consumers/vars metadata, span tiering, and OTLP route handlers end-to-end.

🤖 Generated with Claude Code

prisis and others added 8 commits July 21, 2026 20:48
Extend the observations span store with the Langfuse-style generation
model: a `gen_ai.request.model` span is decoded to `kind: "generation"`
with its model id, prompt/completion token counts, and (opt-in on the
emitter) recorded input/output. Threaded in lockstep through the schema,
`observationInput`, `SpanObservation`/`decodeObservations`, the local
`SpanView`/`ObservationSpan` mirrors (so codegen inlines them), and the
Traces waterfall (a `gen` chip + model/token summary per generation row).

Consumes what the framework now emits (@lunora/ai RAG generation spans,
@lunora/agent OTLP telemetry). costMinor is deferred (needs a price
table). Full span-detail pane lands with the Traces UI pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Add server-side filters to `traces.list` — `errorOnly`, `minDurationMs`,
`functionPath` (substring over the root op), and a `from`/`to` time
window (`to` doubles as a load-older cursor) — via a pure, unit-tested
`matchesTraceFilter`/`filterTraces` (src/telemetry/trace-query.ts) applied
after the fold (before the cap, so a filtered list still fills `limit`).

Add a `by_org_deployment_started` index and push `deploymentId` into the
query, so a deployment-scoped trace list scans that deployment's own spans
rather than filtering the global recent 2000-span window in memory (where
a quiet deployment's older traces fell off the end). Wire a minimal
"errors only" toggle into the Traces tab.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Complete the OTLP ingest contract so any OpenTelemetry SDK/Collector can
ship to the cloud:

- **protobuf** — a Worker-safe, zero-eval hand-rolled OTLP wire decoder
  (src/telemetry/otlp-protobuf.ts) for traces/logs/metrics, since
  protobufjs generates decoders with `new Function` (blocked in Workers).
  `readOtlpBody` branches on Content-Type and decodes protobuf (+ gzip)
  into the same JSON payload shapes; the 415 is gone.
- **/v1/metrics** — decode OTLP metric data points and write them to the
  Analytics-Engine telemetry dataset (a real handler, so the framework's
  `ctx.metrics` POSTs land instead of 404-ing).
- **partialSuccess** — capped batches now return an OTLP
  `partialSuccess { rejectedSpans/rejectedLogRecords/rejectedDataPoints }`
  instead of a silent full-success.

gRPC stays out of scope (Workers can't host a gRPC server).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Harden the OTLP ingest:

- **Scoped ingest key** — a `capability` column on `deployKeys`
  (`deploy` | `ingest`; absent = deploy, backward-compatible). An
  `ingest` key is telemetry-only: `authorizeDeployKey` rejects it from
  the deploy/admin paths, so the token the platform injects into a
  tenant's `otlpSink` (Phase 2) can't be used to deploy. `issue` gained a
  `capability` arg; the keys list surfaces it.
- **Telemetry rate-limit tier** — `/v1/traces|logs|metrics|telemetry`
  now use a separate, generous token bucket keyed on the ingest token
  (per org) instead of the shared per-IP `api` limit, so a busy exporter
  isn't throttled and one noisy tenant can't starve others.
- **PII redaction** — a pure `redactRecord` (src/telemetry/redact.ts)
  scrubs credential-keyed values (authorization, cookie, token, api_key,
  session, …) from span attributes + log fields at decode time.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Wire telemetry into tenants at deploy time (GAPS.md B2's remaining piece):

- **WfP metadata** — `putDispatchScript` now emits `tail_consumers` and
  `plain_text` var bindings (long documented as the provisioner's job but
  never implemented); threaded through `TenantDeploymentSpec`/`provision`.
- **Injection** — the deploy handler sets `LUNORA_OTLP_ENDPOINT` (var),
  `LUNORA_OTLP_TOKEN` (secret), and the `lunora-log-tail` tail consumer,
  so a tenant's `otlpSink` ships to the cloud automatically. Best-effort:
  a resolve failure deploys untelemetered.
- **Scoped ingest token** — one `ingest`-capability key per org (can't
  deploy, per Phase 3), its plaintext stored envelope-encrypted so it's
  re-injected each deploy without re-minting; `ingestKeyCipher`/
  `recordIngestKey` are deploy-key-authorized and race-safe.

Off when `LUNORA_OTLP_ENDPOINT` / the master key are unset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Click any waterfall span to open a detail pane: span id, duration,
status, and — for AI generation spans — the model, prompt/completion
token counts, and the recorded input/output (when the emitter opted in),
plus the span's attributes. Projects `attributes` through `SpanView`/
`ObservationSpan` and the `traces.get` mapping. Row selection clears when
the trace or deployment changes.

Cross-tab links (trace→logs, error span→Issue) remain a follow-on — they
need dashboard tab-state plumbing and the framework carrying traceId into
getIssues.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Add `TelemetryStore.archiveSpans` — every ingested span is streamed to
the telemetry Pipeline (→ R2/Iceberg), tagged `recordType: "span"` +
`organizationId`, so the Traces store scales past D1's hot 48h window.
Wired fire-and-forget into both the `/v1/traces` and `/v1/telemetry`
ingest paths after the D1 write. Mirrors the existing `archiveEvents`;
a pure `spanArchiveRecord` mapper is unit-tested. The R2-SQL read-back
(an action querying the Iceberg table for older traces) stays 🌐, like
the raw-event archive — the write side lands here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Add handler-level tests for the OTLP ingest routes through the real
router (fake action context): missing/invalid bearer → 401, a valid JSON
batch → empty-success, protobuf body now accepted (not 415), a capped
batch → partialSuccess.rejectedSpans, metrics accepted. Update GAPS.md
B2 to record the full OpenTelemetry integration.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
@coderabbitai

coderabbitai Bot commented Jul 21, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)
  • main
  • alpha

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0be2a3d8-a564-406a-a727-3f538d10ff94

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/cloud-otlp-full

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@github-actions

github-actions Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

React Doctor found 1 new issue in 1 file · 1 warning · score 85 / 100 (Great) · 0 fixed · vs feat/cloud-traces-viewer

1 warning

src/client/TracesSection.tsx

  • ⚠️ L176 Unsupported ARIA prop for role role-supports-aria-props

Reviewed by React Doctor for commit c9cc9b3. See inline comments for fixes.

CRITICAL (security):
- Ingest auth was dead-on-arrival: the injected `ingest` key was rejected
  by `authorizeDeployKey` inside `telemetry.ingest`/`logs.ingest` (→ 500),
  so no tenant could ever ingest. Split authorization — `authorizeTelemetryKey`
  accepts ingest+deploy keys for telemetry writes; `authorizeDeployKey` keeps
  barring ingest keys from deploy/admin. `deploy_keys.verify` (the deploy
  entrypoint) also rejects ingest keys, and `LUNORA_OTLP_ENDPOINT` is now
  configured so injection isn't inert.
- Decompression bomb: OTLP bodies now drain through a size-capped reader
  (32 MB decompressed ceiling) instead of buffering unbounded.
- Protobuf varint DoS: capped at 10 bytes, so an all-0xFF body throws (→ 400)
  rather than driving O(n²) BigInt work.
- PII redaction: `token`/`secret` now match as substrings (catching
  `auth_token`/`id_token`/`authToken`/`bearerToken`…), `key` stays scoped to
  credential prefixes so `shard_key` isn't over-redacted.
- a11y: clickable waterfall rows got keyboard (Enter/Space) + role/tabindex.

Refactors (no behavior change):
- Collapsed the three near-identical OTLP handlers into one `withOtlpIngest`
  wrapper (org resolved once — fixes the double-lookup + null-throw).
- Extracted ingest-key provisioning to `src/telemetry/ingest-key.ts`.
- Flattened `decodeObservations`' scattered conditional spreads.
- Collapsed six protobuf resource/scope readers into two generics.
- Deduped the active-ingest-key predicate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
{waterfall.map((row) => (
<div className={`trace-wrow${row.level === "error" ? " trace-wrow-err" : ""}`} key={row.spanId}>
<div
aria-selected={row.spanId === selectedSpanId}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

React Doctor · react-doctor/role-supports-aria-props (warning)

Screen reader users get no help from aria-selected because role button ignores it, so remove it or change the role.

Fix → Only use aria-* attributes that the element's role supports.

Docs

@prisis
prisis merged commit d64ee67 into feat/cloud-traces-viewer Jul 22, 2026
6 checks passed
@prisis
prisis deleted the feat/cloud-otlp-full branch July 22, 2026 05:07
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 9, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant