docs(examples): add Langfuse trace evaluation walkthrough

[Chang Liu (changliu2)]

Summary

• add a customer walkthrough for evaluating conversations already stored in Langfuse without re-running the agent
• reconstruct OpenInference, OTel GenAI, and Langfuse-native observations into ordered ASSERT conversations with tool and retrieval evidence
• include a judge-only eval config and taxonomy with six explicit impermissible categories and one permissible intended behavior
• include a deterministic synthetic corpus and verification script covering 20 traces, 8 sessions, and 80 observations
• add the Langfuse flow to the examples index

Customer flow

1. Export or generate Langfuse trace-detail JSON.
2. Convert it into an ASSERT inference set.
3. Inspect the reconstructed conversations before model spend.
4. Review explicit behavior_categories[].permissible flags.
5. Run only the ASSERT judge stage.
6. Inspect per-conversation verdicts and cited turns.

Validation

• python examples/langfuse_trace_evaluation/verify.py
• Python compilation for all example scripts
• judge-only config load with six configured dimensions
• taxonomy check: 6 impermissible categories, 1 permissible category, 0 missing flags

Scope boundary

The deterministic file-export flow is covered here. Direct API mode uses Langfuse's public v1 trace list/detail endpoints and may require an SDK export on deployments that no longer expose that response shape. Publishing ASSERT judgments back to the original Langfuse sessions is not part of this PR.

[Jake Present (jakepresent)]

The deterministic fixture passes, but the converter drops real conversation turns in a supported/default input shape.

trace_to_spans() tracks seen (role, content) counts across the whole Langfuse session, while each trace recomputes occurrence counts from zero (langfuse_to_assert.py:581-592, shared at :659-677). If a Langfuse session stores one user message per trace and the user sends the same text twice, such as two separate “yes” messages, the second trace’s user turn is treated as already seen and omitted. I reproduced this with two traces in one session, each containing only its current [{role: user, content: yes}] input: the emitted transcript roles are user, assistant, assistant instead of user, assistant, user, assistant. The README explicitly says each user message can be one Langfuse trace, so this corrupts the judge input on the advertised path.

The checked-in verifier only covers fixture traces that replay growing history, and it is not run by any current required check; this PR’s check rollup is CodeQL-only. Please fix the cross-trace deduplication and add the repeated-identical-turn case to an automated test surface that CI executes. The existing fixture verification itself is green: 20 traces become 8 conversations with the expected taxonomy.

[Chang Liu (changliu2)]

Fixed in 4e81b50. I replaced session-wide (role, content) counts with sequence-aware reconciliation: repeated system prefixes and prior-history suffix/incoming-prefix overlap are removed, while assistant outputs anchor later identical user turns as new. Added CI-gated pytest coverage for the two-yes regression, replayed full/suffix histories, reversed trace input, and deterministic ordering. verify.py remains green; the targeted regression plus 182 relevant trace/import tests pass (1 skipped). Ready for re-review.