Repository navigation
feat(agent,ai): emit AI generation spans over OTLP - #160
Conversation
Wrap each `defineRag` embedding-model call in a `ctx.trace` generation span (`gen_ai.operation.name: "embeddings"` + `gen_ai.request.model`) when the bound context exposes `ctx.trace`, so embeds show up on the trace waterfall. Adds an optional `trace?: unknown` to `RagContext` (same decoupling rationale as `auth?`), narrowed to a callable; a hand-built context without it runs embeds untraced, as before. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Add `otlpTelemetry`, the OTLP-over-HTTP counterpart to the Sentry and
Braintrust bridges: it wraps each language-model call and tool execution
in an OTLP span carrying `gen_ai.*` attributes (model, provider, token
usage, tool name) and ships it to a collector, so agent generations land
in the same trace store as the rest of an app's telemetry. Emitted inside
the turn's execution wrapper, so a Workflow replay of a memoized
`step.do('llm:turn:N')` never double-emits. Privacy-safe by default
(`recordInputs`/`recordOutputs` off); fire-and-forget with `waitUntil`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Thank you for following the naming conventions! 🙏 |
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughAdds an OTLP-over-HTTP telemetry integration for model and tool executions, exports it publicly, and adds optional ChangesTelemetry instrumentation
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant LanguageModelOrTool
participant otlpTelemetry
participant OTLPCollector
LanguageModelOrTool->>otlpTelemetry: execute operation and emit gen_ai span
otlpTelemetry->>OTLPCollector: POST span to /v1/traces
otlpTelemetry-->>LanguageModelOrTool: preserve operation result
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/agent/src/telemetry/otlp.ts`:
- Around line 52-67: Make pushAttribute resilient to serialization failures by
handling exceptions from JSON.stringify for object and array values, preventing
errors from escaping into the unhandled emit promise chain. Preserve primitive,
nullish, and successfully serialized attribute behavior; when serialization
fails, skip or safely handle that attribute without throwing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 00ea5329-a30e-43ff-ba79-1e901a63edde
⛔ Files ignored due to path filters (2)
packages/agent/__tests__/telemetry/otlp.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/ai/__tests__/rag.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (4)
packages/agent/src/telemetry/index.tspackages/agent/src/telemetry/otlp.tspackages/ai/src/rag/define-rag.tspackages/ai/src/rag/types.ts
| /** Push one attribute, skipping nullish values and JSON-stringifying non-primitives. */ | ||
| const pushAttribute = (attributes: OtlpAttribute[], key: string, value: unknown): void => { | ||
| if (value === undefined || value === null) { | ||
| return; | ||
| } | ||
|
|
||
| if (typeof value === "boolean" || typeof value === "number" || typeof value === "string") { | ||
| attributes.push(encodeAttribute(key, value satisfies OtlpAttributeValue)); | ||
|
|
||
| return; | ||
| } | ||
|
|
||
| // Objects/arrays (a recorded prompt, tool input) — serialize so the | ||
| // collector stores a queryable string rather than dropping the attribute. | ||
| attributes.push(encodeAttribute(key, JSON.stringify(value))); | ||
| }; |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
JSON.stringify can throw and go unhandled.
pushAttribute JSON-stringifies arbitrary recorded values (prompts, completions, tool inputs) with no try/catch. Since emit (Lines 151-175, 194-206) runs inside promise.then(onFulfilled, onRejected) (Lines 177-184, 208-215) and the resulting promise is never awaited/caught, a circular reference or BigInt in a recorded value (opt-in via recordInputs/recordOutputs) throws synchronously and becomes an unhandled promise rejection — contradicting the doc comment's explicit claim of "no unhandled rejection" (Lines 144-148).
🛡️ Proposed fix
// Objects/arrays (a recorded prompt, tool input) — serialize so the
// collector stores a queryable string rather than dropping the attribute.
- attributes.push(encodeAttribute(key, JSON.stringify(value)));
+ try {
+ attributes.push(encodeAttribute(key, JSON.stringify(value)));
+ } catch {
+ // Circular references / BigInt / etc. — drop rather than crash the caller.
+ }
};📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| /** Push one attribute, skipping nullish values and JSON-stringifying non-primitives. */ | |
| const pushAttribute = (attributes: OtlpAttribute[], key: string, value: unknown): void => { | |
| if (value === undefined || value === null) { | |
| return; | |
| } | |
| if (typeof value === "boolean" || typeof value === "number" || typeof value === "string") { | |
| attributes.push(encodeAttribute(key, value satisfies OtlpAttributeValue)); | |
| return; | |
| } | |
| // Objects/arrays (a recorded prompt, tool input) — serialize so the | |
| // collector stores a queryable string rather than dropping the attribute. | |
| attributes.push(encodeAttribute(key, JSON.stringify(value))); | |
| }; | |
| /** Push one attribute, skipping nullish values and JSON-stringifying non-primitives. */ | |
| const pushAttribute = (attributes: OtlpAttribute[], key: string, value: unknown): void => { | |
| if (value === undefined || value === null) { | |
| return; | |
| } | |
| if (typeof value === "boolean" || typeof value === "number" || typeof value === "string") { | |
| attributes.push(encodeAttribute(key, value satisfies OtlpAttributeValue)); | |
| return; | |
| } | |
| // Objects/arrays (a recorded prompt, tool input) — serialize so the | |
| // collector stores a queryable string rather than dropping the attribute. | |
| try { | |
| attributes.push(encodeAttribute(key, JSON.stringify(value))); | |
| } catch { | |
| // Circular references / BigInt / etc. — drop rather than crash the caller. | |
| } | |
| }; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/agent/src/telemetry/otlp.ts` around lines 52 - 67, Make
pushAttribute resilient to serialization failures by handling exceptions from
JSON.stringify for object and array values, preventing errors from escaping into
the unhandled emit promise chain. Preserve primitive, nullish, and successfully
serialized attribute behavior; when serialization fails, skip or safely handle
that attribute without throwing.
Merging this PR will not alter performance
Comparing Footnotes
|
- @lunora/ai: hoist the resolved embedding model to a local const so the traced `run` closure drops its `as EmbeddingModel` cast. - @lunora/agent: document `otlpTelemetry`'s two deliberate differences from the SDK-backed integrations — no `onError` (span status carries failure) and flat spans (no ambient parent to nest under). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
- observability.mdx: a "Generation spans (AI)" section (gen_ai.* attributes, how @lunora/ai RAG + @lunora/agent otlpTelemetry emit them), and a note that the Lunora cloud ingest accepts protobuf + gzip + partialSuccess and a scoped ingest key, not only Lunora's own JSON sinks. - @lunora/agent docs: the otlpTelemetry integration (per-turn gen_ai spans over OTLP), alongside sentry/braintrust/console. - @lunora/ai docs: RAG embeds are traced as generation spans when ctx.trace is present. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
fead75f to
6f4af8f
Compare
A hosted-cloud monitoring guide (new "Lunora Cloud" nav section): how telemetry reaches the cloud (deploy-time LUNORA_OTLP_ENDPOINT/token + tail_consumers injection), the Traces waterfall + generation spans + span-detail pane, Issues, Logs, metrics, scoped ingest keys, bringing your own OTel SDK/Collector (JSON/protobuf/gzip + partialSuccess), and retention/tiering + PII redaction. Cross-linked with the framework observability docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/docs/src/content/docs/concepts/observability.mdx`:
- Around line 275-293: Update the observability documentation at
apps/docs/src/content/docs/concepts/observability.mdx:275-293 to distinguish
`@lunora/ai` RAG embedding spans as carrying only gen_ai.operation.name and, when
available, gen_ai.request.model, while restricting token usage and
recordInputs/recordOutputs details to `@lunora/agent` model-turn spans. Also
update apps/docs/src/content/docs/cloud/monitoring.mdx:57-62 to state that token
counts and recorded input/output appear only when the emitting span provides
those attributes, so RAG embedding spans may have no token counters.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 7b758456-7c4b-4807-97e5-40607927b326
📒 Files selected for processing (5)
apps/docs/src/content/docs/cloud/monitoring.mdxapps/docs/src/content/docs/concepts/observability.mdxapps/docs/src/content/docs/meta.jsonpackages/agent/docs/index.mdxpackages/ai/docs/index.mdx
| An AI model call is a span with the OpenTelemetry | ||
| [`gen_ai.*`](https://opentelemetry.io/docs/specs/semconv/gen-ai/) attributes on | ||
| it — the model and, once the call returns, its token usage. Two paths emit them: | ||
|
|
||
| - **[`@lunora/ai`](/docs/packages/ai)** traces every `defineRag` embedding call | ||
| as a generation span (`gen_ai.operation.name: "embeddings"`, | ||
| `gen_ai.request.model`) whenever the bound context carries `ctx.trace`. Wrap | ||
| your own `generateText`/`streamText` calls in `ctx.trace("…", fn, { "gen_ai.request.model": model })` | ||
| for the same. | ||
| - **[`@lunora/agent`](/docs/packages/agent)** emits a generation span per model | ||
| turn (and a span per tool call), with token usage, through its `otlpTelemetry` | ||
| integration — add it to `defineAgent({ telemetry: { integrations: [...] } })` | ||
| to opt in. | ||
|
|
||
| A generation span carries `gen_ai.request.model` and, from the model result, | ||
| `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens`; with the agent | ||
| integration's `recordInputs` / `recordOutputs` it also carries the prompt / | ||
| completion (off by default). The Lunora cloud reads these to show the model, | ||
| token counts, and cost on the Traces waterfall. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Distinguish RAG embedding spans from agent generation spans. RAG embeddings emit gen_ai.operation.name and (when available) gen_ai.request.model, but deliberately do not emit token usage or recorded input/output. The current wording promises agent-only fields for every generation span.
apps/docs/src/content/docs/concepts/observability.mdx#L275-L293: state that token usage andrecordInputs/recordOutputsfields apply to@lunora/agentmodel-turn spans; describe RAG embedding spans as model/operation-only.apps/docs/src/content/docs/cloud/monitoring.mdx#L57-L62: clarify that token counts and recorded input/output appear only when the emitting span supplies those attributes; RAG embedding spans may show no token counters.
📍 Affects 2 files
apps/docs/src/content/docs/concepts/observability.mdx#L275-L293(this comment)apps/docs/src/content/docs/cloud/monitoring.mdx#L57-L62
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/src/content/docs/concepts/observability.mdx` around lines 275 -
293, Update the observability documentation at
apps/docs/src/content/docs/concepts/observability.mdx:275-293 to distinguish
`@lunora/ai` RAG embedding spans as carrying only gen_ai.operation.name and, when
available, gen_ai.request.model, while restricting token usage and
recordInputs/recordOutputs details to `@lunora/agent` model-turn spans. Also
update apps/docs/src/content/docs/cloud/monitoring.mdx:57-62 to state that token
counts and recorded input/output appear only when the emitting span provides
those attributes, so RAG embedding spans may have no token counters.
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
What
Completes the framework side of the full-OpenTelemetry integration (Phase 1b of the plan): AI generation spans. The nested-span/
ctx.trace/ctx.metricsmachinery already exists on alpha — this adds the missing generation observations so a downstream OTel collector (the Lunora Cloud, or any collector) gets Langfuse-style model-call spans.@lunora/ai— trace RAG embedsdefineRag's embedding-model call is now wrapped in actx.tracegeneration span (gen_ai.operation.name: "embeddings"+gen_ai.request.model) when the bound context exposesctx.trace.trace?: unknowntoRagContext(mirrors the existingauth?: unknowndecoupling), narrowed to a callable. A hand-built context without it embeds untraced, exactly as before — no behavior change for existing callers.ctx.tracecaptures attributes at span start, and embeddings are cheap — token/cost belong on the LLM-call spans below.@lunora/agent—otlpTelemetryintegrationsentryTelemetry/braintrustTelemetry. Wraps each language-model call and tool execution in an OTLP span withgen_ai.*attributes (model, provider, token usage, tool name) and ships it to a collector.defineAgent({ telemetry: { isEnabled: true, integrations: [otlpTelemetry({ endpoint, token })] } }).step.do('llm:turn:N')never double-emits.recordInputs/recordOutputsoff — only structural metadata + token counts leave the worker); fire-and-forget withwaitUntil; atraceIdoption groups a run's spans into one trace.Why
The Lunora Cloud can render deep trace waterfalls (the framework already emits nested
ctx.tracechild spans with realparentSpanId), but had no AI generation observations. This closes that gap so model calls — cost, tokens, model id — show up in the cloud's Traces view (cloud-side store/UI lands separately).Not included (documented follow-ons)
/v1/metricsingest are cloud-side.Test plan
@lunora/ai: 94 tests pass (incl. 2 new — embeds traced with the model id; untraced withoutctx.trace).@lunora/agent: telemetry suite 37 pass (incl. 6 new forotlpTelemetry— span attrs + token usage as OTLP int strings, sharedtraceId,recordInputsgating, error status, tool spans).tsc --noEmitclean on both packages.🤖 Generated with Claude Code
Summary by CodeRabbit
gen_ai.*generation and tool-call spans, including endpoint normalization, bearer token auth, optional fixed trace IDs, and optionalwaitUntil.partialSuccessbehavior).