feat(eval): add a built-in Langfuse evaluation suite

[benym]

🔎 Existing issue check

• I have searched the existing issues.

🧭 Problem

comet eval currently supports --suite local and --suite langsmith, but it has no built-in Langfuse suite. Teams using Langfuse cannot run the same Comet tasks, treatments, scoring, and reports with first-class remote observability across the selectable Agent CLIs introduced by #249.

✨ Proposed solution

Add Langfuse to the existing suite selector without adding a new subcommand:

comet eval ./comet/eval.yaml --suite langfuse
comet eval ./comet/eval.yaml --suite langfuse --agent codex

• Add langfuse to --suite; keep local as the default.
• Keep --suite orthogonal to the feat(eval): support selectable evaluation agent CLIs #249 --agent selection.
• Do not require new fields in comet/eval.yaml.
• Automatically select the Langfuse optional dependency for this suite so users do not need a separate dependency-install command.
• Keep --collect fully local: no credential check, network request, or plugin download.

Architecture and reuse boundary

Implement eval/langfuse as a thin reporting wrapper around the unchanged local runner:

1. Reuse the local task resolution, treatments, container execution, deterministic validation, Rubric/LLM-judge scoring, Pass@k, Pass^k, quality gates, evidence, and local reports.
2. Authenticate with Langfuse before starting an expensive Agent run.
3. Prepare detailed trajectory support for the Agent selected by feat(eval): support selectable evaluation agent CLIs #249.
4. Upload the core run trace, scores, and experiment summary after local evaluation.
5. Flush the Langfuse client before exit.

Local results remain the only scoring source. Langfuse must not recompute metrics, become a task source, or require creating/synchronizing a Langfuse Dataset.

Configuration

Use the standard Langfuse environment contract:

LANGFUSE_PUBLIC_KEY
LANGFUSE_SECRET_KEY
LANGFUSE_BASE_URL
LANGFUSE_TRACING_ENVIRONMENT

LANGFUSE_BASE_URL supports Langfuse Cloud regions and self-hosted deployments. Comet derives any Agent-specific variables required by the official plugins from these standard values.

Agent trajectory adapters

Agent	Detailed trajectory implementation
Claude Code	Automatically provision and enable the official Langfuse Observability Plugin
Codex CLI	Automatically provision and enable the official Langfuse Codex Plugin
Qoder CLI	Bundle a Comet adapter over Qoder Stop hooks and its JSONL transcript contract

Official references: Claude Code, Codex, and Qoder hooks.

Provision official plugins at pinned versions with integrity metadata under an isolated Eval cache. Enable them through a temporary Agent home/configuration and never modify the user’s global Claude Code, Codex, or Qoder settings. Reuse a valid cache on later runs and protect concurrent provisioning with a lock.

Trace and score mapping

Create one core comet.eval.run trace for each task × treatment × sample. Record:

• task, prompt, treatment, Skill, and profile;
• pass/fail status, deterministic checks, final response, turns, tool calls, duration, tokens, and cost;
• experiment_id, comet_run_id, Agent name/version, and the isolated Subject, Simulator, and Judge session IDs;
• trajectory_status, including an explicit unavailable state when detailed tracing fails.

Attach these run-level scores:

• comet.passed (Boolean);
• comet.checks_pass_rate (numeric, 0–1);
• comet.rubric.<dimension> (numeric, 0–1).

Create one comet.eval.experiment summary trace after the experiment completes. For each task/treatment aggregate, publish comet.pass_at_k, comet.pass_power_k, and the applicable quality-gate result.

Keep the official Agent plugin trajectory hierarchy intact. Link native turn/generation/tool/subagent traces to the core Eval trace through the Agent session IDs rather than rewriting vendor trace trees. Detailed trajectories are diagnostic evidence only and must never affect scoring.

Failure semantics

Use a two-tier contract: core reporting is strict; detailed trajectory reporting is best-effort.

• Missing credentials or failed authentication: stop before Agent execution with actionable setup guidance.
• Eval failure: preserve the local report and still upload the failed run and its available scores.
• Official plugin download/runtime failure or Qoder trajectory conversion failure: continue the Eval, warn once, and set trajectory_status=unavailable.
• Core trace, score, or final flush failure: preserve the local report but return a non-zero CLI status attributed to langfuse_upload.
• Eval and upload failures in the same run: report both; never mask the original Eval failure.
• Never silently downgrade --suite langfuse to local.

Security and privacy

• Pass credentials only through the runtime environment; never persist them in caches, reports, or trace metadata.
• Remove temporary Agent configuration after the run.
• Apply a bounded capture policy to large prompts and tool inputs/outputs and mark truncated values explicitly.
• Treat --suite langfuse as explicit opt-in to remote trace upload.

Relationship to other Eval features

• Depend on feat(eval): support selectable evaluation agent CLIs #249 for Agent selection, isolated role sessions, and normalized run identity.
• Reuse feat(eval): support project-authored task templates in comet/eval.yaml #250 project-authored task templates without adding Langfuse-specific task fields.
• Reuse the generated and frozen task set from feat(eval): automatically generate task templates when none are defined #251 when no task is user-defined.

✅ Acceptance criteria

• --suite langfuse works with every Agent supported by feat(eval): support selectable evaluation agent CLIs #249 and does not add another CLI command.
• The same inputs produce identical local scoring and report results under local and langfuse.
• Langfuse Cloud and self-hosted endpoints are supported through the same environment contract.
• Core run traces, run-level scores, Pass@k, Pass^k, and quality-gate results are queryable by experiment, Agent, task, and treatment.
• Claude Code and Codex expose their official detailed trajectories when their plugins are available.
• Qoder exposes equivalent turn, generation, tool, and Skill observations from its documented transcript format.
• --collect produces no network, credential, or plugin side effects.
• No global Agent configuration is modified.
• Existing local and langsmith behavior and dependency isolation remain unchanged.

🧪 Test plan

• CLI option/help/default tests for local, langsmith, and langfuse, including combinations with --agent.
• Contract tests proving the Langfuse suite reuses local task/treatment/scoring/report behavior.
• Credential, custom base URL, authentication, and --collect no-network tests.
• Pinned plugin provisioning, cache reuse, locking, integrity, cleanup, and global-config preservation tests.
• Claude Code/Codex environment wiring tests and fixed Qoder transcript-to-observation fixtures.
• Score mapping tests for pass/fail, deterministic checks, Rubric dimensions, Pass@k, Pass^k, and quality gates.
• Failure tests for optional trajectory loss, strict core upload/flush errors, local-report preservation, and simultaneous Eval/upload errors.
• CI uses fake Langfuse clients and fixed transcript fixtures; it does not require live cloud credentials.

🎯 Primary area

Eval (app, domains/eval, and eval suite adapters)

🪐 Workflow phase

Not phase-specific

🔀 Alternatives considered

1. Local reports only: does not provide supported remote observability.
2. Synchronize Comet tasks into Langfuse Datasets: gives native Dataset Runs but introduces remote task ownership, synchronization, versioning, and cleanup that are unnecessary for this feature.
3. Require users to preinstall every Agent plugin: reduces implementation work but makes the suite harder to use and less reproducible.
4. Reimplement every Agent trajectory in Comet: duplicates official Claude Code and Codex integrations and creates avoidable maintenance drift.

🧰 Compatibility notes

This is an additive --suite langfuse value. Existing task/manifest contracts, report formats, local and langsmith suites, and credential behavior remain unchanged. The Langfuse dependency and integration code must stay isolated from the local suite.

🧩 Out of scope

• Using Langfuse as a Task or Dataset source.
• Automatically creating or updating Langfuse Datasets.
• Running Langfuse-hosted evaluators instead of Comet validation/Rubric logic.
• Adding new Langfuse-specific CLI commands or required manifest fields.