Telemetry-grounded, calibrated failure attribution for agent oversight.
When a multi-agent run fails, oversight needs to answer which agent, which step — and know when to trust that answer. The Who&When benchmark (Zhang et al., ICML 2025) shows the usual approach (an LLM reading the transcript) tops out around 53.5% agent / 14.2% step. Every baseline is a single LLM judge over a bespoke transcript.
This repo asks a different question: how much of "where did it go wrong" is already sitting in the OpenTelemetry GenAI spans your agent stack emits — tool spans with ERROR status, retries, stalled orchestrators — and can we attach a calibrated confidence an overseer can act on?
| Method | Agent acc. | Step acc. | Inference cost |
|---|---|---|---|
| Random | 25.9 | 9.5 | — |
| Structural (telemetry features, no LLM) | 51.1 | 17.9 | none |
| Judge: gpt-4o-mini / llama-3.3-70b / claude-3-haiku (re-run) | 47.3 / 47.3 / 48.4 | 21.2 / 25.5 / 23.4 | API |
A model-free attributor beats every re-run LLM judge on agent attribution (51.1 vs 47-48) at zero inference cost, is competitive on steps, and — unlike a bare judge — reports a calibrated confidence. And the judges are unreliable: three of them agree on the culprit agent only 54.5% of the time and the decisive step 32.6%; one model re-run at temp 0.7 disagrees with itself on the step ~25% of the time. Confidence enables selective oversight:
| Coverage (by confidence) | Agent acc. | Step acc. |
|---|---|---|
| top 10% | 77.8 | 50.0 |
| top 50% | 56.5 | 25.0 |
| 100% | 51.1 | 17.9 |
See paper/paper.md and paper/figures/.
structural.py— deterministic step/agent features + logistic heads (no LLM).otel.py— bridge to OpenTelemetry GenAI agent/tool spans (so the same features run on real production traces), plus feature recomputation from spans.calibration.py— ECE + risk–coverage (selective prediction).judge.py/reliability.py/hybrid.py— LLM-judge reproduction, inter-model / run-to-run reliability, and a confidence-gated hybrid router (needs an OpenRouter key).eval.py— the Who&When evaluation protocol (agent + step) with cross-split and k-fold.
pip install -e .
bash scripts/get_data.sh # clones the Who&When benchmark
python scripts/run_all.py # structural results + calibration + selective (no API)
python scripts/make_figures.py # figures
OPENROUTER_API_KEY=... python scripts/run_judge.py # judge reliability + hybrid (optional)Trajectory monitoring is a core, fragile lever for overseeing agents. This shows attribution can be grounded in the telemetry standard agents already emit, done at zero inference cost, and — unlike a bare judge — reports a calibrated confidence so a monitor can flag only what it can attribute reliably.
Built on prior work: llm-trace-cost-attribution
(OTel GenAI trace parsing) and llm-judge-calibration
(judge reliability).
Apache-2.0. The Who&When benchmark is © its authors (ag2ai/Agents_Failure_Attribution); this
repo does not vendor it — scripts/get_data.sh fetches it.