Repository navigation
feat(do,server): SpanHandle.recordEvaluation for gen_ai.evaluation attributes - #208
Conversation
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
|
Thank you for following the naming conventions! 🙏 |
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (11)
📒 Files selected for processing (5)
WalkthroughAdds the public ChangesEvaluation telemetry
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant DispatchSpan
participant SpanCollector
Caller->>DispatchSpan: recordEvaluation(evaluation)
DispatchSpan->>SpanCollector: forward evaluation
SpanCollector->>SpanCollector: merge evaluation attributes
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Merging this PR will improve performance by 11.78%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | count, no attributes |
64.7 µs | 57.6 µs | +12.31% |
| ⚡ | 1 shard × 1000 rows (single round-trip) |
3.2 ms | 2.9 ms | +11.24% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing feat/span-record-evaluation (cd18ace) with alpha (d0236f2)2
Footnotes
-
2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
alpha(5caabb3) during the generation of this report, so d0236f2 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
…tributes WIP snapshot of the evaluation-span work, committed on its original base (alpha.100) before rebasing, so the pre-rebase state stays recoverable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rfaces Alpha grew three things this work predates, so the rebased feature needed adapting rather than replaying: - ctx.span's lazy SpanHandle proxy in shard-do.ts now delegates recordEvaluation, so the wide event gets it alongside ctx.trace. - @lunora/testing's recording double records the gen_ai.evaluation.* attributes (via the package's own evaluationAttributes, not the shared/ copy — packages/server pins rootDir, so shared/ is not importable there). - SpanHandle test doubles across the server suite gained the member. Also refreshes the api snapshots and retargets the runtime test at the current otlpSpanBody signature (one arg, returns the bare span; the resourceSpans envelope moved a tier up). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d53e841 to
cd18ace
Compare
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Closes the last missing piece of the AI evaluation-telemetry feature: nothing in production code could emit
gen_ai.evaluation.*attributes onto a generation span.The gap
The feature exists in three parts. Two were already merged:
gen_ai.evaluation.*off generation spansapps/cloud/src/telemetry/otlp.ts(EVALUATION_PREFIX)packages/testing/src/evaluation-telemetry.ts@lunora/do+@lunora/serverGrepping alpha for
recordEvaluationandgen_ai.evaluationreturns hits only underpackages/testing. So the cloud console has a decoder parsing eval attributes off real generation spans, but the only thing that ever produced them was the test harness — a scorer's grade from a real run never reached the trace.What this adds
SpanHandle.recordEvaluation({ name, score, label? })→ emitsgen_ai.evaluation.<name>.score(number) and, when given,gen_ai.evaluation.<name>.label(string).shared/evaluation-attributes.ts— bundler-inlined key builder, zero-dependency per theshared/convention. Sanitizes the name to[A-Za-z0-9._-]so a scorer calledcontains:shippedstill yields a well-formed attribute key. Throws on an empty name or non-finite score rather than poisoning the grade downstream.@lunora/do— wired intocreateSpanCollector, so it merges in like any other post-hoc attribute and flushes over OTLP with no special casing.@lunora/server— declaresSpanEvaluationstructurally rather than importing the shared type, becausepackages/serverpinsrootDir: "."and cannot reference out-of-package files.Privacy-safe by construction: only the name, score, and optional label are emitted — never the graded prompt or completion.
Available on both span surfaces
Because alpha had already refactored the handle into the shared
createSpanCollector,recordEvaluationlands on bothctx.tracespans and the per-dispatch wide event (ctx.span). The original draft only reachedctx.trace.Tests
@lunora/do— the pair lands on the recorded span; non-finite score rejects.@lunora/runtime— survives OTLP encoding verbatim: an integer score encodes asintValue: "1", a fractional one asdoubleValue: 0.8, and the label asstringValue. This is the hop the cloud decoder depends on.@lunora/testing— the recording double records the same keys production does, so harness assertions can't pass against a narrower shape than what ships.Provenance
Written against
alpha.100and rebased ~150 commits onto current alpha. Three conflicts were genuine, not textual:SpanHandlegrewaddEvent/addLink/recordExceptionupstream — all kept,recordEvaluationinserted in alphabetical position.spanHandleliteral incontext-telemetry.tsbecame thecreateSpanCollectorfactory — the method was ported into the factory rather than either side taken verbatim.otlpSpanBodychanged to one argument returning the bare span (theresourceSpansenvelope moved a tier up), so the runtime test was retargeted.Split into two commits: the original work, then the adaptation, so the rebase decisions are reviewable on their own.
🤖 Generated with Claude Code
Summary by CodeRabbit
gen_ai.evaluation.*attributes.SpanEvaluationtype for integration use.