Skip to content

feat(agent,ai,testing): llm session id + eval score telemetry - #173

Merged
prisis merged 3 commits into
alphafrom
feat/llm-evals-sessions
Jul 22, 2026
Merged

prisis merged 3 commits into
alphafrom
feat/llm-evals-sessions

Conversation

@prisis

@prisis prisis commented Jul 22, 2026 •

Copy link
Copy Markdown
Member

Tier 1 — LLM sessions + evals (framework emit side)

  • Sessions/threads: generation spans now carry gen_ai.conversation.id — auto-threaded from an agent run's threadKey (@lunora/agent) and settable on RAG embeds via RagContext.conversationId. Absent unless set (backward-compatible).
  • Evals: new recordEvaluation({ name, score, label?, span? }) + evaluationAttributes(...) in @lunora/testing — emits gen_ai.evaluation.<name>.score (+ optional .label), attachable to a generation span via the feat(server,do,ai): post-hoc span attributes + traced rag embed cost #170 SpanHandle or returned for standalone use. Privacy-safe (name/score/label only).

Additive; the cloud side (feat/cloud-sessions-evals) decodes these exact attribute names.

Verify: agent 285 / ai 109 / testing 86 tests pass; tsc clean; touched files eslint-clean.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added conversation and session identifiers to AI generation and embedding telemetry.
    • Added evaluation telemetry utilities for recording scores and optional labels on generation spans.
    • Exported evaluation telemetry tools through the testing package.
  • Bug Fixes
    • Preserved existing telemetry output when no conversation identifier is provided.
    • Added validation for missing evaluation names and invalid scores.

Sessions/threads: tag generation spans with a conversation/session id so a
multi-turn conversation groups in the trace store.
- @lunora/agent: otlpTelemetry gains `conversationId`, emitted as
  `gen_ai.conversation.id` on the language-model span; absent by default. The
  agent workflow auto-threads the run's `threadKey` as the conversation id.
- @lunora/ai RAG: RagContext gains an optional `conversationId`, emitted as
  `gen_ai.conversation.id` on each embed generation span when set.

Evals: @lunora/testing gains `recordEvaluation`/`evaluationAttributes`, turning
an eval verdict into `gen_ai.evaluation.<name>.score` (+ optional `.label`)
attributes — attachable to a generation span via the post-hoc SpanHandle or
usable standalone. Privacy-safe: only name/score/label are emitted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197KjhmBDB3PXAc6TXiPqZ9
@netlify

netlify Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit e072705
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a60f9ab2716d00008097d65
😎 Deploy Preview https://deploy-preview-173--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Review Change Stack

Walkthrough

The change adds optional conversation IDs to generation and embedding telemetry, propagates workflow thread keys into OTLP configuration, and introduces testing utilities that validate, construct, attach, and export evaluation telemetry attributes.

Changes

Conversation telemetry

Layer / File(s) Summary
Conversation span contracts and emission
packages/agent/src/telemetry/otlp.ts, packages/ai/src/rag/types.ts, packages/ai/src/rag/define-rag.ts
Optional conversation IDs are added to telemetry contexts and emitted as gen_ai.conversation.id on generation and embedding spans when present.
Workflow conversation propagation
packages/agent/src/workflow.ts
Workflow threadKey values are passed into automatically configured OTLP telemetry as conversation IDs.

Evaluation telemetry

Layer / File(s) Summary
Evaluation attribute construction
packages/testing/src/evaluation-telemetry.ts
Evaluation inputs are validated, names are sanitized, and score and optional label attributes are constructed.
Evaluation recording and public exports
packages/testing/src/evaluation-telemetry.ts, packages/testing/src/index.ts
Attributes can be attached to optional spans and returned for standalone emission, with the new APIs exported publicly.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Workflow
  participant OTLPTelemetry
  participant GenerationSpan
  participant EmbeddingSpan
  Workflow->>OTLPTelemetry: pass threadKey as conversationId
  OTLPTelemetry->>GenerationSpan: set gen_ai.conversation.id
  Workflow->>EmbeddingSpan: provide RagContext conversationId
  EmbeddingSpan->>EmbeddingSpan: set gen_ai.conversation.id
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description gives a useful high-level summary, but it does not follow the required template sections or include linked issues, checklist, notes, or CLA text. Add the template sections (Summary, Linked issues, Test plan, Checklist, Notes, CLA) and include the required CLA statement.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise, conventional, and clearly summarizes the main change: LLM session ID and evaluation telemetry across agent, AI, and testing.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/llm-evals-sessions

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/agent/src/telemetry/otlp.ts`:
- Around line 202-204: Update the conversation ID handling in the telemetry
attribute construction to normalize an empty conversationId to undefined before
passing it to pushAttribute. Preserve non-empty IDs and the existing omission
behavior for nullish values, matching the established normalization used by
defineRag.

In `@packages/testing/src/evaluation-telemetry.ts`:
- Around line 63-68: Update sanitizeName so distinct evaluation names cannot map
to the same sanitized key; replace unsafe characters with a deterministic
injective encoding that preserves safe characters, or reject names containing
unrepresentable characters. Ensure names such as contains:shipped and
contains/shipped remain distinct when used as telemetry keys.
- Around line 90-91: Update the label handling in the evaluation telemetry
attribute construction to runtime-validate that input.label is a string before
assigning it to attributes; reject null, numbers, objects, and other non-string
values, and explicitly preserve or enforce the intended empty-string policy
consistent with existing validation for name and score.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 1d276e0c-114b-4962-aa29-353eb09f08a4

📥 Commits

Reviewing files that changed from the base of the PR and between f94b04b and 4d9d7bf.

⛔ Files ignored due to path filters (4)
  • api-snapshots/testing.api.md is excluded by none and included by none
  • packages/agent/__tests__/telemetry/otlp.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/ai/__tests__/rag.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/testing/__tests__/evaluation-telemetry.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (6)
  • packages/agent/src/telemetry/otlp.ts
  • packages/agent/src/workflow.ts
  • packages/ai/src/rag/define-rag.ts
  • packages/ai/src/rag/types.ts
  • packages/testing/src/evaluation-telemetry.ts
  • packages/testing/src/index.ts

Comment on lines +202 to +204
// Session/thread grouping — absent unless a conversation id was set,
// so a run with no session id emits exactly as before.
pushAttribute(attributes, "gen_ai.conversation.id", conversationId);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Treat empty conversation IDs as unset.

pushAttribute skips nullish values, not empty strings, so conversationId: "" emits an empty gen_ai.conversation.id. Normalize empty IDs to undefined, matching packages/ai/src/rag/define-rag.ts Lines 328-334.

Proposed fix
-                pushAttribute(attributes, "gen_ai.conversation.id", conversationId);
+                pushAttribute(
+                    attributes,
+                    "gen_ai.conversation.id",
+                    conversationId?.length ? conversationId : undefined,
+                );
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Session/thread grouping — absent unless a conversation id was set,
// so a run with no session id emits exactly as before.
pushAttribute(attributes, "gen_ai.conversation.id", conversationId);
// Session/thread grouping — absent unless a conversation id was set,
// so a run with no session id emits exactly as before.
pushAttribute(
attributes,
"gen_ai.conversation.id",
conversationId?.length ? conversationId : undefined,
);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/agent/src/telemetry/otlp.ts` around lines 202 - 204, Update the
conversation ID handling in the telemetry attribute construction to normalize an
empty conversationId to undefined before passing it to pushAttribute. Preserve
non-empty IDs and the existing omission behavior for nullish values, matching
the established normalization used by defineRag.

Comment on lines +63 to +68
const sanitizeName = (name: string): string => {
let out = "";

for (const char of name) {
out += SAFE_NAME_CHAR.test(char) ? char : "_";
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Make evaluation-name sanitization collision-free.

Replacing every unsafe character with _ makes distinct names such as contains:shipped and contains/shipped produce the same key. When both evaluations are attached to one span, the later score or label can overwrite the earlier telemetry. Use an injective encoding for unsafe characters or reject names that cannot be represented without collisions.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/testing/src/evaluation-telemetry.ts` around lines 63 - 68, Update
sanitizeName so distinct evaluation names cannot map to the same sanitized key;
replace unsafe characters with a deterministic injective encoding that preserves
safe characters, or reject names containing unrepresentable characters. Ensure
names such as contains:shipped and contains/shipped remain distinct when used as
telemetry keys.

Comment on lines +90 to +91
if (input.label !== undefined) {
attributes[`gen_ai.evaluation.${key}.label`] = input.label;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Validate label at runtime before emitting it.

Unlike name and score, label is only TypeScript-validated. A JavaScript caller or as any value can pass null, a number, or an object, which is then forwarded to setAttributes as an invalid attribute value. Reject non-string labels (and explicitly decide whether empty strings are allowed).

Proposed validation
     if (typeof input.score !== "number" || !Number.isFinite(input.score)) {
         throw new LunoraError("BAD_REQUEST", "`@lunora/testing`: recordEvaluation `score` must be a finite number");
     }

+    if (input.label !== undefined && typeof input.label !== "string") {
+        throw new LunoraError("BAD_REQUEST", "`@lunora/testing`: recordEvaluation `label` must be a string");
+    }
+
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if (input.label !== undefined) {
attributes[`gen_ai.evaluation.${key}.label`] = input.label;
if (typeof input.score !== "number" || !Number.isFinite(input.score)) {
throw new LunoraError("BAD_REQUEST", "`@lunora/testing`: recordEvaluation `score` must be a finite number");
}
if (input.label !== undefined && typeof input.label !== "string") {
throw new LunoraError("BAD_REQUEST", "`@lunora/testing`: recordEvaluation `label` must be a string");
}
if (input.label !== undefined) {
attributes[`gen_ai.evaluation.${key}.label`] = input.label;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/testing/src/evaluation-telemetry.ts` around lines 90 - 91, Update
the label handling in the evaluation telemetry attribute construction to
runtime-validate that input.label is a string before assigning it to attributes;
reject null, numbers, objects, and other non-string values, and explicitly
preserve or enforce the intended empty-string policy consistent with existing
validation for name and score.

@prisis
prisis merged commit 33838f9 into alpha Jul 22, 2026
12 of 13 checks passed
@prisis
prisis deleted the feat/llm-evals-sessions branch July 22, 2026 17:11
@codspeed

codspeed Bot commented Jul 22, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 160 untouched benchmarks
⏩ 1 skipped benchmark1


Comparing feat/llm-evals-sessions (e072705) with alpha (cace8e5)2

Open in CodSpeed

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

  2. No successful run was found on alpha (3698e52) during the generation of this report, so cace8e5 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

prisis added a commit that referenced this pull request Jul 28, 2026
…, otp reset

From better-auth-ui #173, #162, #211 and #337, checked against our code.

The password minimum was hard-coded to 8, which is a guess at the server's
rule: too low and the user is told 'too short' only after a round-trip, too
high and we refuse passwords the server would take. It is config now, meant
to be matched to emailAndPassword, and every password field routes through
it. <PasswordStrength> shows the rules as a checklist rather than a bar,
because a bar says a password is unacceptable without saying what to change.

The providers dropped the field. AuthUIConfig.password existed and
resolveContext honoured it, but all five rebuilt their config object
field-by-field and omitted it — so a configured policy was unreachable and
the checklist always showed the default. Regression test added.

Username availability now uses /is-username-available, which better-auth
exposed all along: a taken name surfaced only as a failed sign-up. It is
debounced and drops out-of-order answers, and reports 'unknown' rather than
'taken' on failure so it can never block a sign-up the server would allow.

Forgot-password can use the emailOTP endpoints. An app that recovers by code
got 'Reset password isn't enabled' from /request-password-reset, which names
neither cause nor fix.

Sign-up seeds email/name from the link, and the invitation bounce now carries
the invited address — retyping it is how an invitee ends up creating a
different account the invitation doesn't match. Passwords are never
prefilled: an allow-list, because a password in a query string lands in
history, referrers and logs.

Also organization.showSlug, and the last-used badge now covers email and
magic-link — better-auth records those too, so badging only social buttons
made the feature invisible for the commonest route.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — e072705c Deployed Jul 22, 2026 by prisis via Benchmarks #880
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant