Skip to content

docs: add eve evals reporter integration (LAM-1855)#186

Open
laminar-coding-agent[bot] wants to merge 2 commits into
mainfrom
docs/lam-1855-eve-reporter
Open

docs: add eve evals reporter integration (LAM-1855)#186
laminar-coding-agent[bot] wants to merge 2 commits into
mainfrom
docs/lam-1855-eve-reporter

Conversation

@laminar-coding-agent

@laminar-coding-agent laminar-coding-agent Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • New evaluations/eve page documenting the LaminarReporter for eve evals: how each eval maps to a Laminar datapoint, gate vs. soft score encoding, registering the reporter in evals.config.ts, and reading/querying results.
  • Two screenshots (list + detail) captured from a verified support-triage run against the local prod stack, including a genuine gate failure.
  • Changelog entry under June 2026 and the docs.json nav entry (Evaluations group).

Notes

  • Page content was re-verified against the shipped LaminarReporter (lmnr-ts feat/lam-1855-eve-laminar-reporter, commit 1f6c6e6): the datapoint input is the eval id, and metadata carries verdict, status, session id, model id, tool-call names, and failed assertions.
  • Validated with npx mintlify broken-links (clean); no em dashes or banned marketing words.

Test plan

  • Mintlify preview renders the page under Evaluations → eve reporter
  • Both images display at readable size inside their <Frame>
  • Internal links resolve (comparing-runs, concepts, manual-evaluation, sql-editor, cli, mcp)

Note

Low Risk
Documentation-only changes (MDX, nav, changelog); no application or SDK code in this diff.

Overview
Adds documentation for shipping eve eval runs to Laminar via LaminarReporter: mapping (eval → datapoint, gates as gate:<name> scores, soft assertions, executor output), setup in evals/evals.config.ts, running eve eval, and reading/querying results in the UI, SQL, CLI, and MCP. Includes list and detail screenshots and links to comparing runs and related eval docs.

Navigation: new Evaluations → eve reporter page (evaluations/eve) in docs.json.

Changelog: July 2026 entry for the eve reporter feature.

Reviewed by Cursor Bugbot for commit 32f99e3. Bugbot is set up for automated code reviews on this repo. Configure here.

Document the LaminarReporter for eve.dev evals: how each eval maps to a
Laminar datapoint, gate vs. soft score encoding, registering the reporter
in evals.config.ts, and reading/querying the results. Includes list and
detail screenshots from a verified support-triage run, a changelog entry,
and the docs.json nav entry.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
laminarai 🟢 Ready View Preview Jun 29, 2026, 9:34 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@laminar-coding-agent

Copy link
Copy Markdown
Contributor Author

Re-verified this page against eve 0.22.6 (it was written against 0.16.2) and a real end-to-end run of the companion SDK PR (lmnr-ai/lmnr-ts#266) against a local Laminar stack. No changes needed:

  • evals/evals.config.ts registration with defineEvalConfig({ reporters: [...] }) works verbatim on 0.22.6.
  • The per-eval reporters field on defineEval still exists in 0.22.6 types, so the "scope a destination to a single eval" note holds.
  • The console-output excerpt matches 0.22.6's console reporter format (verdict glyph, gates n/m, soft name: % entries).
  • Gate scores land as binary gate:<name> and soft scores as numeric, exactly as the "How it maps" section describes — confirmed in ClickHouse evaluation_datapoints and in the UI (group progression chart + per-score columns).
  • npx mintlify broken-links passes; nav (docs.json) and changelog entries are in place.

Resolve changelog conflict: eve reporter entry moves to July 2026 since
that is when the feature ships.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@laminar-coding-agent

Copy link
Copy Markdown
Contributor Author

Re-verified this page against eve 0.22.6 (latest) and the implementation in lmnr-ai/lmnr-ts#266:

  • Registration snippet (evals/evals.config.ts, defineEvalConfig, reporters: [new LaminarReporter(...)]) matches eve 0.22.6 — ran it in a real eve project end-to-end against a local Laminar stack.
  • Score encoding described on the page (gate assertions → binary gate:<name>, soft assertions → numeric) matches what actually lands in evaluation_datapoints.
  • The SQL example (simpleJSONExtractFloat(scores, 'gate:succeeded')) verified working via the SQL API (the map-subscript alternative fails on the current deployment, so this is the right form).
  • The env-var config path (LMNR_BASE_URL for self-hosted) mentioned here had a client-side port-resolution bug — fixed in LAM-1855: add Laminar reporter for eve evals lmnr-ts#266 (commit 70ae486), so the docs are accurate as written.
  • docs.json nav entry, changelog entry, and both images present; mintlify broken-links passes.

No changes needed on this branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant