Repository navigation
feat(cloud): full OpenTelemetry integration (generations, protobuf, hardened ingest, tiering) - #161
Conversation
Extend the observations span store with the Langfuse-style generation model: a `gen_ai.request.model` span is decoded to `kind: "generation"` with its model id, prompt/completion token counts, and (opt-in on the emitter) recorded input/output. Threaded in lockstep through the schema, `observationInput`, `SpanObservation`/`decodeObservations`, the local `SpanView`/`ObservationSpan` mirrors (so codegen inlines them), and the Traces waterfall (a `gen` chip + model/token summary per generation row). Consumes what the framework now emits (@lunora/ai RAG generation spans, @lunora/agent OTLP telemetry). costMinor is deferred (needs a price table). Full span-detail pane lands with the Traces UI pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Add server-side filters to `traces.list` — `errorOnly`, `minDurationMs`, `functionPath` (substring over the root op), and a `from`/`to` time window (`to` doubles as a load-older cursor) — via a pure, unit-tested `matchesTraceFilter`/`filterTraces` (src/telemetry/trace-query.ts) applied after the fold (before the cap, so a filtered list still fills `limit`). Add a `by_org_deployment_started` index and push `deploymentId` into the query, so a deployment-scoped trace list scans that deployment's own spans rather than filtering the global recent 2000-span window in memory (where a quiet deployment's older traces fell off the end). Wire a minimal "errors only" toggle into the Traces tab. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Complete the OTLP ingest contract so any OpenTelemetry SDK/Collector can
ship to the cloud:
- **protobuf** — a Worker-safe, zero-eval hand-rolled OTLP wire decoder
(src/telemetry/otlp-protobuf.ts) for traces/logs/metrics, since
protobufjs generates decoders with `new Function` (blocked in Workers).
`readOtlpBody` branches on Content-Type and decodes protobuf (+ gzip)
into the same JSON payload shapes; the 415 is gone.
- **/v1/metrics** — decode OTLP metric data points and write them to the
Analytics-Engine telemetry dataset (a real handler, so the framework's
`ctx.metrics` POSTs land instead of 404-ing).
- **partialSuccess** — capped batches now return an OTLP
`partialSuccess { rejectedSpans/rejectedLogRecords/rejectedDataPoints }`
instead of a silent full-success.
gRPC stays out of scope (Workers can't host a gRPC server).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Harden the OTLP ingest: - **Scoped ingest key** — a `capability` column on `deployKeys` (`deploy` | `ingest`; absent = deploy, backward-compatible). An `ingest` key is telemetry-only: `authorizeDeployKey` rejects it from the deploy/admin paths, so the token the platform injects into a tenant's `otlpSink` (Phase 2) can't be used to deploy. `issue` gained a `capability` arg; the keys list surfaces it. - **Telemetry rate-limit tier** — `/v1/traces|logs|metrics|telemetry` now use a separate, generous token bucket keyed on the ingest token (per org) instead of the shared per-IP `api` limit, so a busy exporter isn't throttled and one noisy tenant can't starve others. - **PII redaction** — a pure `redactRecord` (src/telemetry/redact.ts) scrubs credential-keyed values (authorization, cookie, token, api_key, session, …) from span attributes + log fields at decode time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Wire telemetry into tenants at deploy time (GAPS.md B2's remaining piece): - **WfP metadata** — `putDispatchScript` now emits `tail_consumers` and `plain_text` var bindings (long documented as the provisioner's job but never implemented); threaded through `TenantDeploymentSpec`/`provision`. - **Injection** — the deploy handler sets `LUNORA_OTLP_ENDPOINT` (var), `LUNORA_OTLP_TOKEN` (secret), and the `lunora-log-tail` tail consumer, so a tenant's `otlpSink` ships to the cloud automatically. Best-effort: a resolve failure deploys untelemetered. - **Scoped ingest token** — one `ingest`-capability key per org (can't deploy, per Phase 3), its plaintext stored envelope-encrypted so it's re-injected each deploy without re-minting; `ingestKeyCipher`/ `recordIngestKey` are deploy-key-authorized and race-safe. Off when `LUNORA_OTLP_ENDPOINT` / the master key are unset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Click any waterfall span to open a detail pane: span id, duration, status, and — for AI generation spans — the model, prompt/completion token counts, and the recorded input/output (when the emitter opted in), plus the span's attributes. Projects `attributes` through `SpanView`/ `ObservationSpan` and the `traces.get` mapping. Row selection clears when the trace or deployment changes. Cross-tab links (trace→logs, error span→Issue) remain a follow-on — they need dashboard tab-state plumbing and the framework carrying traceId into getIssues. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Add `TelemetryStore.archiveSpans` — every ingested span is streamed to the telemetry Pipeline (→ R2/Iceberg), tagged `recordType: "span"` + `organizationId`, so the Traces store scales past D1's hot 48h window. Wired fire-and-forget into both the `/v1/traces` and `/v1/telemetry` ingest paths after the D1 write. Mirrors the existing `archiveEvents`; a pure `spanArchiveRecord` mapper is unit-tested. The R2-SQL read-back (an action querying the Iceberg table for older traces) stays 🌐, like the raw-event archive — the write side lands here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Add handler-level tests for the OTLP ingest routes through the real router (fake action context): missing/invalid bearer → 401, a valid JSON batch → empty-success, protobuf body now accepted (not 415), a capped batch → partialSuccess.rejectedSpans, metrics accepted. Update GAPS.md B2 to record the full OpenTelemetry integration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for following the naming conventions! 🙏 |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
|
React Doctor found 1 new issue in 1 file · 1 warning · score 85 / 100 (Great) · 0 fixed · vs 1 warning
Reviewed by React Doctor for commit |
CRITICAL (security): - Ingest auth was dead-on-arrival: the injected `ingest` key was rejected by `authorizeDeployKey` inside `telemetry.ingest`/`logs.ingest` (→ 500), so no tenant could ever ingest. Split authorization — `authorizeTelemetryKey` accepts ingest+deploy keys for telemetry writes; `authorizeDeployKey` keeps barring ingest keys from deploy/admin. `deploy_keys.verify` (the deploy entrypoint) also rejects ingest keys, and `LUNORA_OTLP_ENDPOINT` is now configured so injection isn't inert. - Decompression bomb: OTLP bodies now drain through a size-capped reader (32 MB decompressed ceiling) instead of buffering unbounded. - Protobuf varint DoS: capped at 10 bytes, so an all-0xFF body throws (→ 400) rather than driving O(n²) BigInt work. - PII redaction: `token`/`secret` now match as substrings (catching `auth_token`/`id_token`/`authToken`/`bearerToken`…), `key` stays scoped to credential prefixes so `shard_key` isn't over-redacted. - a11y: clickable waterfall rows got keyboard (Enter/Space) + role/tabindex. Refactors (no behavior change): - Collapsed the three near-identical OTLP handlers into one `withOtlpIngest` wrapper (org resolved once — fixes the double-lookup + null-throw). - Extracted ingest-key provisioning to `src/telemetry/ingest-key.ts`. - Flattened `decodeObservations`' scattered conditional spreads. - Collapsed six protobuf resource/scope readers into two generics. - Deduped the active-ingest-key predicate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
| {waterfall.map((row) => ( | ||
| <div className={`trace-wrow${row.level === "error" ? " trace-wrow-err" : ""}`} key={row.spanId}> | ||
| <div | ||
| aria-selected={row.spanId === selectedSpanId} |
There was a problem hiding this comment.
React Doctor · react-doctor/role-supports-aria-props (warning)
Screen reader users get no help from aria-selected because role button ignores it, so remove it or change the role.
Fix → Only use aria-* attributes that the element's role supports.
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
What
Completes the full-OpenTelemetry integration on the cloud side, stacked on the Traces work (#157). Pairs with framework #160 (AI generation-span emission → alpha). All 9 planned phases landed.
bc335622observationsgainskind:"generation"(model, prompt/completion tokens, opt-in input/output); decoder promotes anygen_ai.request.modelspan; waterfall shows agenchipd67d6000traces.listgainserrorOnly/minDurationMs/functionPath/from/to; aby_org_deployment_startedindex fixes quiet deployments' older traces vanishing past the 2000-span windowec918166/v1/metrics→ Analytics Engine,partialSuccesson capae89b149ingestdeploy-key capability (can't deploy), per-org telemetry rate-limit tier, key-based PII redactione3b7361dLUNORA_OTLP_ENDPOINT/token +tail_consumers; per-org ingest key stored envelope-encrypted for re-injectionb0c9f1c221b8752aarchiveSpansstreams spans to Pipeline → R2/Iceberg (scales past D1)540dfea6Why
The Traces viewer (#157) had a full span store but the value was bottlenecked and the ingest incomplete. This makes the cloud a real, hardened OpenTelemetry endpoint: any SDK/Collector ships over JSON or protobuf, safely (scoped key, rate-limit, redaction); AI generations show cost/tokens; traces filter/paginate and tier to columnar storage; and telemetry flows automatically once a tenant is deployed.
Key discovery: the framework's deep-span stack (
ctx.trace,ctx.metrics) was already on alpha — so/v1/metricsingest became load-bearing and generations became the framework's only missing emit (shipped in #160).Not included (🌐 / follow-on, tracked in GAPS.md B2)
LUNORA_OTLP_ENDPOINTset.traceIdintogetIssues.Test plan
pnpm --filter @lunora/cloud run test),tsc --noEmitclean, codegen regenerated.tail_consumers/vars metadata, span tiering, and OTLP route handlers end-to-end.🤖 Generated with Claude Code