Skip to content

feat(do,server): SpanHandle.recordEvaluation for gen_ai.evaluation attributes - #208

Merged
prisis merged 2 commits into
alphafrom
feat/span-record-evaluation
Jul 27, 2026
Merged

prisis merged 2 commits into
alphafrom
feat/span-record-evaluation

Conversation

@prisis

@prisis prisis commented Jul 27, 2026 •

Copy link
Copy Markdown
Member

Closes the last missing piece of the AI evaluation-telemetry feature: nothing in production code could emit gen_ai.evaluation.* attributes onto a generation span.

The gap

The feature exists in three parts. Two were already merged:

Part Where Status before this PR
Cloud decoder — reads gen_ai.evaluation.* off generation spans apps/cloud/src/telemetry/otlp.ts (EVALUATION_PREFIX) merged
Test-time emitter packages/testing/src/evaluation-telemetry.ts merged (alpha)
Production span emitter @lunora/do + @lunora/server missing

Grepping alpha for recordEvaluation and gen_ai.evaluation returns hits only under packages/testing. So the cloud console has a decoder parsing eval attributes off real generation spans, but the only thing that ever produced them was the test harness — a scorer's grade from a real run never reached the trace.

What this adds

SpanHandle.recordEvaluation({ name, score, label? }) → emits gen_ai.evaluation.<name>.score (number) and, when given, gen_ai.evaluation.<name>.label (string).

  • shared/evaluation-attributes.ts — bundler-inlined key builder, zero-dependency per the shared/ convention. Sanitizes the name to [A-Za-z0-9._-] so a scorer called contains:shipped still yields a well-formed attribute key. Throws on an empty name or non-finite score rather than poisoning the grade downstream.
  • @lunora/do — wired into createSpanCollector, so it merges in like any other post-hoc attribute and flushes over OTLP with no special casing.
  • @lunora/server — declares SpanEvaluation structurally rather than importing the shared type, because packages/server pins rootDir: "." and cannot reference out-of-package files.

Privacy-safe by construction: only the name, score, and optional label are emitted — never the graded prompt or completion.

Available on both span surfaces

Because alpha had already refactored the handle into the shared createSpanCollector, recordEvaluation lands on both ctx.trace spans and the per-dispatch wide event (ctx.span). The original draft only reached ctx.trace.

Tests

  • @lunora/do — the pair lands on the recorded span; non-finite score rejects.
  • @lunora/runtime — survives OTLP encoding verbatim: an integer score encodes as intValue: "1", a fractional one as doubleValue: 0.8, and the label as stringValue. This is the hop the cloud decoder depends on.
  • @lunora/testing — the recording double records the same keys production does, so harness assertions can't pass against a narrower shape than what ships.
do 1253 · server 451 · testing 115 · runtime 759   all passing
tsc --noEmit  61 projects clean
eslint        clean
api:check     matches all 39 snapshots

Provenance

Written against alpha.100 and rebased ~150 commits onto current alpha. Three conflicts were genuine, not textual:

  1. SpanHandle grew addEvent / addLink / recordException upstream — all kept, recordEvaluation inserted in alphabetical position.
  2. The inline spanHandle literal in context-telemetry.ts became the createSpanCollector factory — the method was ported into the factory rather than either side taken verbatim.
  3. otlpSpanBody changed to one argument returning the bare span (the resourceSpans envelope moved a tier up), so the runtime test was retargeted.

Split into two commits: the original work, then the adaptation, so the rebase decisions are reviewable on their own.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added support for recording AI evaluation results on generation/dispatch span telemetry.
    • Evaluation metric name, numeric score, and optional label are captured as standardized gen_ai.evaluation.* attributes.
    • Exposed a new SpanEvaluation type for integration use.
    • Updated the testing harness so evaluation attributes are recorded and can be asserted after execution.

@netlify

netlify Bot commented Jul 27, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit cd18ace
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a675227c3d7330008465ba8
😎 Deploy Preview https://deploy-preview-208--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@coderabbitai

coderabbitai Bot commented Jul 27, 2026 •

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e5a69ba0-26b2-4659-ac1b-00bee21cc0eb

📥 Commits

Reviewing files that changed from the base of the PR and between d53e841 and cd18ace.

⛔ Files ignored due to path filters (11)
  • api-snapshots/do.api.md is excluded by none and included by none
  • api-snapshots/lunora.api.md is excluded by none and included by none
  • api-snapshots/server.api.md is excluded by none and included by none
  • packages/do/__tests__/tracing.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/runtime/__tests__/otlp-evaluation.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/server/__tests__/context.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/server/__tests__/functions-v-from.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/server/__tests__/functions.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/server/__tests__/otel.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • shared/evaluation-attributes.ts is excluded by none and included by none
  • shared/span-event.ts is excluded by none and included by none
📒 Files selected for processing (5)
  • packages/do/src/context-telemetry.ts
  • packages/do/src/shard-do.ts
  • packages/server/src/index.ts
  • packages/server/src/types.ts
  • packages/testing/src/harness.ts

Walkthrough

Adds the public SpanEvaluation contract and SpanHandle.recordEvaluation, propagates evaluations through Durable Object dispatch spans, records normalized evaluation attributes, and updates the testing harness.

Changes

Evaluation telemetry

Layer / File(s) Summary
Public evaluation contract
packages/server/src/types.ts, packages/server/src/index.ts
Defines and exports SpanEvaluation, and adds recordEvaluation to SpanHandle.
Collector and dispatch propagation
packages/do/src/context-telemetry.ts, packages/do/src/shard-do.ts
Forwards evaluations from dispatch spans and merges normalized gen_ai.evaluation.* attributes into collected span attributes.
Test harness support
packages/testing/src/harness.ts
Adds no-op and recording implementations of recordEvaluation, storing converted attributes on recorded spans.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant DispatchSpan
  participant SpanCollector
  Caller->>DispatchSpan: recordEvaluation(evaluation)
  DispatchSpan->>SpanCollector: forward evaluation
  SpanCollector->>SpanCollector: merge evaluation attributes
Loading

Possibly related PRs

Suggested labels: package: do, package: server, package: testing

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the new SpanHandle.recordEvaluation support for gen_ai.evaluation attributes.
Description check ✅ Passed The description covers the feature, rationale, implementation, and tests; only the linked issues and explicit CLA text are missing.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/span-record-evaluation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codspeed

codspeed Bot commented Jul 27, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 11.78%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
✅ 237 untouched benchmarks
⏩ 2 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ count, no attributes 64.7 µs 57.6 µs +12.31%
⚡ 1 shard × 1000 rows (single round-trip) 3.2 ms 2.9 ms +11.24%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing feat/span-record-evaluation (cd18ace) with alpha (d0236f2)2

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on alpha (5caabb3) during the generation of this report, so d0236f2 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

prisis and others added 2 commits July 27, 2026 14:42
…tributes

WIP snapshot of the evaluation-span work, committed on its original base
(alpha.100) before rebasing, so the pre-rebase state stays recoverable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rfaces

Alpha grew three things this work predates, so the rebased feature needed
adapting rather than replaying:

- ctx.span's lazy SpanHandle proxy in shard-do.ts now delegates
  recordEvaluation, so the wide event gets it alongside ctx.trace.
- @lunora/testing's recording double records the gen_ai.evaluation.*
  attributes (via the package's own evaluationAttributes, not the shared/
  copy — packages/server pins rootDir, so shared/ is not importable there).
- SpanHandle test doubles across the server suite gained the member.

Also refreshes the api snapshots and retargets the runtime test at the
current otlpSpanBody signature (one arg, returns the bare span; the
resourceSpans envelope moved a tier up).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@prisis
prisis force-pushed the feat/span-record-evaluation branch from d53e841 to cd18ace Compare July 27, 2026 12:42
@prisis
prisis merged commit 3623d7c into alpha Jul 27, 2026
14 of 16 checks passed
@prisis
prisis deleted the feat/span-record-evaluation branch July 27, 2026 12:42
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — cd18ace0 Deployed Jul 27, 2026 by prisis via Benchmarks #1065
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant