Skip to content

feat(agent,ai): emit AI generation spans over OTLP - #160

Merged
prisis merged 6 commits into
alphafrom
feat/framework-otlp-spans
Jul 22, 2026
Merged

prisis merged 6 commits into
alphafrom
feat/framework-otlp-spans

Conversation

@prisis

@prisis prisis commented Jul 21, 2026 •

Copy link
Copy Markdown
Member

What

Completes the framework side of the full-OpenTelemetry integration (Phase 1b of the plan): AI generation spans. The nested-span/ctx.trace/ctx.metrics machinery already exists on alpha — this adds the missing generation observations so a downstream OTel collector (the Lunora Cloud, or any collector) gets Langfuse-style model-call spans.

@lunora/ai — trace RAG embeds

  • defineRag's embedding-model call is now wrapped in a ctx.trace generation span (gen_ai.operation.name: "embeddings" + gen_ai.request.model) when the bound context exposes ctx.trace.
  • Adds an optional trace?: unknown to RagContext (mirrors the existing auth?: unknown decoupling), narrowed to a callable. A hand-built context without it embeds untraced, exactly as before — no behavior change for existing callers.
  • Token usage is deliberately not on the embed span: ctx.trace captures attributes at span start, and embeddings are cheap — token/cost belong on the LLM-call spans below.

@lunora/agent — otlpTelemetry integration

  • New OTLP-over-HTTP counterpart to sentryTelemetry/braintrustTelemetry. Wraps each language-model call and tool execution in an OTLP span with gen_ai.* attributes (model, provider, token usage, tool name) and ships it to a collector.
  • Plugs into defineAgent({ telemetry: { isEnabled: true, integrations: [otlpTelemetry({ endpoint, token })] } }).
  • Emitted inside the turn's execution wrapper, so a Workflow replay of a memoized step.do('llm:turn:N') never double-emits.
  • Privacy-safe by default (recordInputs/recordOutputs off — only structural metadata + token counts leave the worker); fire-and-forget with waitUntil; a traceId option groups a run's spans into one trace.

Why

The Lunora Cloud can render deep trace waterfalls (the framework already emits nested ctx.trace child spans with real parentSpanId), but had no AI generation observations. This closes that gap so model calls — cost, tokens, model id — show up in the cloud's Traces view (cloud-side store/UI lands separately).

Not included (documented follow-ons)

  • gzip on the OTLP POST (marginal for single-span posts; finicky stream bodies in Workers).
  • OTLP protobuf/gRPC transports and /v1/metrics ingest are cloud-side.

Test plan

  • @lunora/ai: 94 tests pass (incl. 2 new — embeds traced with the model id; untraced without ctx.trace).
  • @lunora/agent: telemetry suite 37 pass (incl. 6 new for otlpTelemetry — span attrs + token usage as OTLP int strings, shared traceId, recordInputs gating, error status, tool spans).
  • tsc --noEmit clean on both packages.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added experimental OTLP-over-HTTP telemetry integration that emits gen_ai.* generation and tool-call spans, including endpoint normalization, bearer token auth, optional fixed trace IDs, and optional waitUntil.
    • Added optional tracing for RAG embedding generation when trace support is present in the RAG action context, including embedding model metadata.
  • Documentation
    • Added hosted observability/monitoring documentation for cloud ingest, generation span conventions, and OTLP HTTP support (including gzip and partialSuccess behavior).
    • Updated agent and RAG docs to reflect the new tracing/OTLP behavior and integration options.

prisis and others added 2 commits July 21, 2026 20:39
Wrap each `defineRag` embedding-model call in a `ctx.trace` generation
span (`gen_ai.operation.name: "embeddings"` + `gen_ai.request.model`)
when the bound context exposes `ctx.trace`, so embeds show up on the
trace waterfall. Adds an optional `trace?: unknown` to `RagContext`
(same decoupling rationale as `auth?`), narrowed to a callable; a
hand-built context without it runs embeds untraced, as before.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
Add `otlpTelemetry`, the OTLP-over-HTTP counterpart to the Sentry and
Braintrust bridges: it wraps each language-model call and tool execution
in an OTLP span carrying `gen_ai.*` attributes (model, provider, token
usage, tool name) and ships it to a collector, so agent generations land
in the same trace store as the rest of an app's telemetry. Emitted inside
the turn's execution wrapper, so a Workflow replay of a memoized
`step.do('llm:turn:N')` never double-emits. Privacy-safe by default
(`recordInputs`/`recordOutputs` off); fire-and-forget with `waitUntil`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
@netlify

netlify Bot commented Jul 21, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit 3242e04
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a6050a444804b0008b19671
😎 Deploy Preview https://deploy-preview-160--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@coderabbitai

coderabbitai Bot commented Jul 21, 2026 •

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 47f83fbd-c8d5-4de4-b644-3ce7929974c2

📥 Commits

Reviewing files that changed from the base of the PR and between afaaa4e and 3242e04.

📒 Files selected for processing (1)
  • apps/docs/src/content/docs/concepts/observability.mdx

Walkthrough

Adds an OTLP-over-HTTP telemetry integration for model and tool executions, exports it publicly, and adds optional gen_ai tracing around RAG embedding generation when a trace function is supplied. Documentation covers cloud monitoring, ingest, and observability behavior.

Changes

Telemetry instrumentation

Layer / File(s) Summary
OTLP telemetry integration
packages/agent/src/telemetry/otlp.ts, packages/agent/src/telemetry/index.ts
Defines OTLP options, emits model and tool spans through /v1/traces, and publicly exports the integration and its options.
RAG embedding tracing
packages/ai/src/rag/types.ts, packages/ai/src/rag/define-rag.ts
Adds optional RagContext.trace support and wraps embedding calls with gen_ai operation and model attributes when available.
Observability documentation
apps/docs/src/content/docs/cloud/monitoring.mdx, apps/docs/src/content/docs/concepts/observability.mdx, apps/docs/src/content/docs/meta.json, packages/agent/docs/index.mdx, packages/ai/docs/index.mdx
Documents generation spans, OTLP integrations, cloud monitoring, ingest formats, authentication, retention, and RAG tracing behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LanguageModelOrTool
  participant otlpTelemetry
  participant OTLPCollector
  LanguageModelOrTool->>otlpTelemetry: execute operation and emit gen_ai span
  otlpTelemetry->>OTLPCollector: POST span to /v1/traces
  otlpTelemetry-->>LanguageModelOrTool: preserve operation result
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise, conventional, and matches the main change: adding OTLP emission for AI generation spans.
Description check ✅ Passed It covers the change, rationale, and test plan, but omits template sections like Linked issues and Checklist.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/framework-otlp-spans

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/agent/src/telemetry/otlp.ts`:
- Around line 52-67: Make pushAttribute resilient to serialization failures by
handling exceptions from JSON.stringify for object and array values, preventing
errors from escaping into the unhandled emit promise chain. Preserve primitive,
nullish, and successfully serialized attribute behavior; when serialization
fails, skip or safely handle that attribute without throwing.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 00ea5329-a30e-43ff-ba79-1e901a63edde

📥 Commits

Reviewing files that changed from the base of the PR and between 67ca641 and d9e554f.

⛔ Files ignored due to path filters (2)
  • packages/agent/__tests__/telemetry/otlp.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/ai/__tests__/rag.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (4)
  • packages/agent/src/telemetry/index.ts
  • packages/agent/src/telemetry/otlp.ts
  • packages/ai/src/rag/define-rag.ts
  • packages/ai/src/rag/types.ts

Comment on lines +52 to +67
/** Push one attribute, skipping nullish values and JSON-stringifying non-primitives. */
const pushAttribute = (attributes: OtlpAttribute[], key: string, value: unknown): void => {
if (value === undefined || value === null) {
return;
}

if (typeof value === "boolean" || typeof value === "number" || typeof value === "string") {
attributes.push(encodeAttribute(key, value satisfies OtlpAttributeValue));

return;
}

// Objects/arrays (a recorded prompt, tool input) — serialize so the
// collector stores a queryable string rather than dropping the attribute.
attributes.push(encodeAttribute(key, JSON.stringify(value)));
};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

JSON.stringify can throw and go unhandled.

pushAttribute JSON-stringifies arbitrary recorded values (prompts, completions, tool inputs) with no try/catch. Since emit (Lines 151-175, 194-206) runs inside promise.then(onFulfilled, onRejected) (Lines 177-184, 208-215) and the resulting promise is never awaited/caught, a circular reference or BigInt in a recorded value (opt-in via recordInputs/recordOutputs) throws synchronously and becomes an unhandled promise rejection — contradicting the doc comment's explicit claim of "no unhandled rejection" (Lines 144-148).

🛡️ Proposed fix
     // Objects/arrays (a recorded prompt, tool input) — serialize so the
     // collector stores a queryable string rather than dropping the attribute.
-    attributes.push(encodeAttribute(key, JSON.stringify(value)));
+    try {
+        attributes.push(encodeAttribute(key, JSON.stringify(value)));
+    } catch {
+        // Circular references / BigInt / etc. — drop rather than crash the caller.
+    }
 };
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
/** Push one attribute, skipping nullish values and JSON-stringifying non-primitives. */
const pushAttribute = (attributes: OtlpAttribute[], key: string, value: unknown): void => {
if (value === undefined || value === null) {
return;
}
if (typeof value === "boolean" || typeof value === "number" || typeof value === "string") {
attributes.push(encodeAttribute(key, value satisfies OtlpAttributeValue));
return;
}
// Objects/arrays (a recorded prompt, tool input) — serialize so the
// collector stores a queryable string rather than dropping the attribute.
attributes.push(encodeAttribute(key, JSON.stringify(value)));
};
/** Push one attribute, skipping nullish values and JSON-stringifying non-primitives. */
const pushAttribute = (attributes: OtlpAttribute[], key: string, value: unknown): void => {
if (value === undefined || value === null) {
return;
}
if (typeof value === "boolean" || typeof value === "number" || typeof value === "string") {
attributes.push(encodeAttribute(key, value satisfies OtlpAttributeValue));
return;
}
// Objects/arrays (a recorded prompt, tool input) — serialize so the
// collector stores a queryable string rather than dropping the attribute.
try {
attributes.push(encodeAttribute(key, JSON.stringify(value)));
} catch {
// Circular references / BigInt / etc. — drop rather than crash the caller.
}
};
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/agent/src/telemetry/otlp.ts` around lines 52 - 67, Make
pushAttribute resilient to serialization failures by handling exceptions from
JSON.stringify for object and array values, preventing errors from escaping into
the unhandled emit promise chain. Preserve primitive, nullish, and successfully
serialized attribute behavior; when serialization fails, skip or safely handle
that attribute without throwing.

@codspeed

codspeed Bot commented Jul 21, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 4 untouched benchmarks
⏩ 157 skipped benchmarks1


Comparing feat/framework-otlp-spans (3242e04) with alpha (67ca641)2

Open in CodSpeed

Footnotes

  1. 157 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on alpha (65969dc) during the generation of this report, so 67ca641 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

- @lunora/ai: hoist the resolved embedding model to a local const so the
  traced `run` closure drops its `as EmbeddingModel` cast.
- @lunora/agent: document `otlpTelemetry`'s two deliberate differences from
  the SDK-backed integrations — no `onError` (span status carries failure)
  and flat spans (no ambient parent to nest under).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
- observability.mdx: a "Generation spans (AI)" section (gen_ai.* attributes,
  how @lunora/ai RAG + @lunora/agent otlpTelemetry emit them), and a note that
  the Lunora cloud ingest accepts protobuf + gzip + partialSuccess and a scoped
  ingest key, not only Lunora's own JSON sinks.
- @lunora/agent docs: the otlpTelemetry integration (per-turn gen_ai spans over
  OTLP), alongside sentry/braintrust/console.
- @lunora/ai docs: RAG embeds are traced as generation spans when ctx.trace is
  present.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
@prisis
prisis force-pushed the feat/framework-otlp-spans branch from fead75f to 6f4af8f Compare July 21, 2026 22:32
A hosted-cloud monitoring guide (new "Lunora Cloud" nav section): how
telemetry reaches the cloud (deploy-time LUNORA_OTLP_ENDPOINT/token +
tail_consumers injection), the Traces waterfall + generation spans +
span-detail pane, Issues, Logs, metrics, scoped ingest keys, bringing
your own OTel SDK/Collector (JSON/protobuf/gzip + partialSuccess), and
retention/tiering + PII redaction. Cross-linked with the framework
observability docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
@prisis
prisis merged commit 4c7f0cb into alpha Jul 22, 2026
11 of 12 checks passed
@prisis
prisis deleted the feat/framework-otlp-spans branch July 22, 2026 05:10

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@apps/docs/src/content/docs/concepts/observability.mdx`:
- Around line 275-293: Update the observability documentation at
apps/docs/src/content/docs/concepts/observability.mdx:275-293 to distinguish
`@lunora/ai` RAG embedding spans as carrying only gen_ai.operation.name and, when
available, gen_ai.request.model, while restricting token usage and
recordInputs/recordOutputs details to `@lunora/agent` model-turn spans. Also
update apps/docs/src/content/docs/cloud/monitoring.mdx:57-62 to state that token
counts and recorded input/output appear only when the emitting span provides
those attributes, so RAG embedding spans may have no token counters.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7b758456-7c4b-4807-97e5-40607927b326

📥 Commits

Reviewing files that changed from the base of the PR and between 8fe8d8b and afaaa4e.

📒 Files selected for processing (5)
  • apps/docs/src/content/docs/cloud/monitoring.mdx
  • apps/docs/src/content/docs/concepts/observability.mdx
  • apps/docs/src/content/docs/meta.json
  • packages/agent/docs/index.mdx
  • packages/ai/docs/index.mdx

Comment on lines +275 to +293
An AI model call is a span with the OpenTelemetry
[`gen_ai.*`](https://opentelemetry.io/docs/specs/semconv/gen-ai/) attributes on
it — the model and, once the call returns, its token usage. Two paths emit them:

- **[`@lunora/ai`](/docs/packages/ai)** traces every `defineRag` embedding call
as a generation span (`gen_ai.operation.name: "embeddings"`,
`gen_ai.request.model`) whenever the bound context carries `ctx.trace`. Wrap
your own `generateText`/`streamText` calls in `ctx.trace("…", fn, { "gen_ai.request.model": model })`
for the same.
- **[`@lunora/agent`](/docs/packages/agent)** emits a generation span per model
turn (and a span per tool call), with token usage, through its `otlpTelemetry`
integration — add it to `defineAgent({ telemetry: { integrations: [...] } })`
to opt in.

A generation span carries `gen_ai.request.model` and, from the model result,
`gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens`; with the agent
integration's `recordInputs` / `recordOutputs` it also carries the prompt /
completion (off by default). The Lunora cloud reads these to show the model,
token counts, and cost on the Traces waterfall.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Distinguish RAG embedding spans from agent generation spans. RAG embeddings emit gen_ai.operation.name and (when available) gen_ai.request.model, but deliberately do not emit token usage or recorded input/output. The current wording promises agent-only fields for every generation span.

  • apps/docs/src/content/docs/concepts/observability.mdx#L275-L293: state that token usage and recordInputs/recordOutputs fields apply to @lunora/agent model-turn spans; describe RAG embedding spans as model/operation-only.
  • apps/docs/src/content/docs/cloud/monitoring.mdx#L57-L62: clarify that token counts and recorded input/output appear only when the emitting span supplies those attributes; RAG embedding spans may show no token counters.
📍 Affects 2 files
  • apps/docs/src/content/docs/concepts/observability.mdx#L275-L293 (this comment)
  • apps/docs/src/content/docs/cloud/monitoring.mdx#L57-L62
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/docs/src/content/docs/concepts/observability.mdx` around lines 275 -
293, Update the observability documentation at
apps/docs/src/content/docs/concepts/observability.mdx:275-293 to distinguish
`@lunora/ai` RAG embedding spans as carrying only gen_ai.operation.name and, when
available, gen_ai.request.model, while restricting token usage and
recordInputs/recordOutputs details to `@lunora/agent` model-turn spans. Also
update apps/docs/src/content/docs/cloud/monitoring.mdx:57-62 to state that token
counts and recorded input/output appear only when the emitting span provides
those attributes, so RAG embedding spans may have no token counters.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 9, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — 3242e046 Deployed Jul 22, 2026 by prisis via Benchmarks #826
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant