Repository navigation
feat(agent,ai,testing): llm session id + eval score telemetry - #173
Conversation
Sessions/threads: tag generation spans with a conversation/session id so a multi-turn conversation groups in the trace store. - @lunora/agent: otlpTelemetry gains `conversationId`, emitted as `gen_ai.conversation.id` on the language-model span; absent by default. The agent workflow auto-threads the run's `threadKey` as the conversation id. - @lunora/ai RAG: RagContext gains an optional `conversationId`, emitted as `gen_ai.conversation.id` on each embed generation span when set. Evals: @lunora/testing gains `recordEvaluation`/`evaluationAttributes`, turning an eval verdict into `gen_ai.evaluation.<name>.score` (+ optional `.label`) attributes — attachable to a generation span via the post-hoc SpanHandle or usable standalone. Privacy-safe: only name/score/label are emitted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
WalkthroughThe change adds optional conversation IDs to generation and embedding telemetry, propagates workflow thread keys into OTLP configuration, and introduces testing utilities that validate, construct, attach, and export evaluation telemetry attributes. ChangesConversation telemetry
Evaluation telemetry
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Workflow
participant OTLPTelemetry
participant GenerationSpan
participant EmbeddingSpan
Workflow->>OTLPTelemetry: pass threadKey as conversationId
OTLPTelemetry->>GenerationSpan: set gen_ai.conversation.id
Workflow->>EmbeddingSpan: provide RagContext conversationId
EmbeddingSpan->>EmbeddingSpan: set gen_ai.conversation.id
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
|
Thank you for following the naming conventions! 🙏 |
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/agent/src/telemetry/otlp.ts`:
- Around line 202-204: Update the conversation ID handling in the telemetry
attribute construction to normalize an empty conversationId to undefined before
passing it to pushAttribute. Preserve non-empty IDs and the existing omission
behavior for nullish values, matching the established normalization used by
defineRag.
In `@packages/testing/src/evaluation-telemetry.ts`:
- Around line 63-68: Update sanitizeName so distinct evaluation names cannot map
to the same sanitized key; replace unsafe characters with a deterministic
injective encoding that preserves safe characters, or reject names containing
unrepresentable characters. Ensure names such as contains:shipped and
contains/shipped remain distinct when used as telemetry keys.
- Around line 90-91: Update the label handling in the evaluation telemetry
attribute construction to runtime-validate that input.label is a string before
assigning it to attributes; reject null, numbers, objects, and other non-string
values, and explicitly preserve or enforce the intended empty-string policy
consistent with existing validation for name and score.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 1d276e0c-114b-4962-aa29-353eb09f08a4
⛔ Files ignored due to path filters (4)
api-snapshots/testing.api.mdis excluded by none and included by nonepackages/agent/__tests__/telemetry/otlp.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/ai/__tests__/rag.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/testing/__tests__/evaluation-telemetry.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (6)
packages/agent/src/telemetry/otlp.tspackages/agent/src/workflow.tspackages/ai/src/rag/define-rag.tspackages/ai/src/rag/types.tspackages/testing/src/evaluation-telemetry.tspackages/testing/src/index.ts
| // Session/thread grouping — absent unless a conversation id was set, | ||
| // so a run with no session id emits exactly as before. | ||
| pushAttribute(attributes, "gen_ai.conversation.id", conversationId); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Treat empty conversation IDs as unset.
pushAttribute skips nullish values, not empty strings, so conversationId: "" emits an empty gen_ai.conversation.id. Normalize empty IDs to undefined, matching packages/ai/src/rag/define-rag.ts Lines 328-334.
Proposed fix
- pushAttribute(attributes, "gen_ai.conversation.id", conversationId);
+ pushAttribute(
+ attributes,
+ "gen_ai.conversation.id",
+ conversationId?.length ? conversationId : undefined,
+ );📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Session/thread grouping — absent unless a conversation id was set, | |
| // so a run with no session id emits exactly as before. | |
| pushAttribute(attributes, "gen_ai.conversation.id", conversationId); | |
| // Session/thread grouping — absent unless a conversation id was set, | |
| // so a run with no session id emits exactly as before. | |
| pushAttribute( | |
| attributes, | |
| "gen_ai.conversation.id", | |
| conversationId?.length ? conversationId : undefined, | |
| ); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/agent/src/telemetry/otlp.ts` around lines 202 - 204, Update the
conversation ID handling in the telemetry attribute construction to normalize an
empty conversationId to undefined before passing it to pushAttribute. Preserve
non-empty IDs and the existing omission behavior for nullish values, matching
the established normalization used by defineRag.
| const sanitizeName = (name: string): string => { | ||
| let out = ""; | ||
|
|
||
| for (const char of name) { | ||
| out += SAFE_NAME_CHAR.test(char) ? char : "_"; | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Make evaluation-name sanitization collision-free.
Replacing every unsafe character with _ makes distinct names such as contains:shipped and contains/shipped produce the same key. When both evaluations are attached to one span, the later score or label can overwrite the earlier telemetry. Use an injective encoding for unsafe characters or reject names that cannot be represented without collisions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/testing/src/evaluation-telemetry.ts` around lines 63 - 68, Update
sanitizeName so distinct evaluation names cannot map to the same sanitized key;
replace unsafe characters with a deterministic injective encoding that preserves
safe characters, or reject names containing unrepresentable characters. Ensure
names such as contains:shipped and contains/shipped remain distinct when used as
telemetry keys.
| if (input.label !== undefined) { | ||
| attributes[`gen_ai.evaluation.${key}.label`] = input.label; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Validate label at runtime before emitting it.
Unlike name and score, label is only TypeScript-validated. A JavaScript caller or as any value can pass null, a number, or an object, which is then forwarded to setAttributes as an invalid attribute value. Reject non-string labels (and explicitly decide whether empty strings are allowed).
Proposed validation
if (typeof input.score !== "number" || !Number.isFinite(input.score)) {
throw new LunoraError("BAD_REQUEST", "`@lunora/testing`: recordEvaluation `score` must be a finite number");
}
+ if (input.label !== undefined && typeof input.label !== "string") {
+ throw new LunoraError("BAD_REQUEST", "`@lunora/testing`: recordEvaluation `label` must be a string");
+ }
+📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if (input.label !== undefined) { | |
| attributes[`gen_ai.evaluation.${key}.label`] = input.label; | |
| if (typeof input.score !== "number" || !Number.isFinite(input.score)) { | |
| throw new LunoraError("BAD_REQUEST", "`@lunora/testing`: recordEvaluation `score` must be a finite number"); | |
| } | |
| if (input.label !== undefined && typeof input.label !== "string") { | |
| throw new LunoraError("BAD_REQUEST", "`@lunora/testing`: recordEvaluation `label` must be a string"); | |
| } | |
| if (input.label !== undefined) { | |
| attributes[`gen_ai.evaluation.${key}.label`] = input.label; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/testing/src/evaluation-telemetry.ts` around lines 90 - 91, Update
the label handling in the evaluation telemetry attribute construction to
runtime-validate that input.label is a string before assigning it to attributes;
reject null, numbers, objects, and other non-string values, and explicitly
preserve or enforce the intended empty-string policy consistent with existing
validation for name and score.
Merging this PR will not alter performance
Comparing Footnotes
|
…, otp reset From better-auth-ui #173, #162, #211 and #337, checked against our code. The password minimum was hard-coded to 8, which is a guess at the server's rule: too low and the user is told 'too short' only after a round-trip, too high and we refuse passwords the server would take. It is config now, meant to be matched to emailAndPassword, and every password field routes through it. <PasswordStrength> shows the rules as a checklist rather than a bar, because a bar says a password is unacceptable without saying what to change. The providers dropped the field. AuthUIConfig.password existed and resolveContext honoured it, but all five rebuilt their config object field-by-field and omitted it — so a configured policy was unreachable and the checklist always showed the default. Regression test added. Username availability now uses /is-username-available, which better-auth exposed all along: a taken name surfaced only as a failed sign-up. It is debounced and drops out-of-order answers, and reports 'unknown' rather than 'taken' on failure so it can never block a sign-up the server would allow. Forgot-password can use the emailOTP endpoints. An app that recovers by code got 'Reset password isn't enabled' from /request-password-reset, which names neither cause nor fix. Sign-up seeds email/name from the link, and the invitation bounce now carries the invited address — retyping it is how an invitee ends up creating a different account the invitation doesn't match. Passwords are never prefilled: an allow-list, because a password in a query string lands in history, referrers and logs. Also organization.showSlug, and the last-used badge now covers email and magic-link — better-auth records those too, so badging only social buttons made the feature invisible for the commonest route. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Tier 1 — LLM sessions + evals (framework emit side)
gen_ai.conversation.id— auto-threaded from an agent run'sthreadKey(@lunora/agent) and settable on RAG embeds viaRagContext.conversationId. Absent unless set (backward-compatible).recordEvaluation({ name, score, label?, span? })+evaluationAttributes(...)in@lunora/testing— emitsgen_ai.evaluation.<name>.score(+ optional.label), attachable to a generation span via the feat(server,do,ai): post-hoc span attributes + traced rag embed cost #170SpanHandleor returned for standalone use. Privacy-safe (name/score/label only).Additive; the cloud side (
feat/cloud-sessions-evals) decodes these exact attribute names.Verify: agent 285 / ai 109 / testing 86 tests pass; tsc clean; touched files eslint-clean.
🤖 Generated with Claude Code
Summary by CodeRabbit