From 7b71d5f563e73226e76dbbdc429ad6e8348c54e7 Mon Sep 17 00:00:00 2001 From: "my.nguyen" Date: Tue, 17 Nov 2026 13:50:55 +0700 Subject: [PATCH 1/5] feat(gooddata-eval): label harness errors; opt-in inherited date filter QA-30213, phase 1: a failing run is labelled ERROR only when the test harness failed (its simulated user or its judge), FAIL otherwise. The rule is one fail-closed allowlist in core/outcome.py; chat_error, agent_timeout, turn_incomplete and any unknown cause stay FAIL. The labels decide nothing: an ERROR run still counts as a run that did not pass, so gates, pass rates and scores are unchanged. The labels appear in the "Failing runs" part of the failure message, in failed_runs[] (exit_reason, outcome, failed_checks), on the raised exception (outcome when every run errored, and shadow, the verdict a phase-2 candidate rule would give, recorded only for QA-30296) and in Langfuse as an extra eval_error score. A timed-out or incomplete turn now records agent_timeout / turn_incomplete instead of chat_error. QA-30214: with GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER or --accept-context-date-filter, the visualization filter check accepts a date filter carried over from the dashboard in user_context when the expected output has none. jira: QA-30213 risk: low Co-Authored-By: Claude Opus 5.5 --- packages/gooddata-eval/AGENTS.md | 2 + packages/gooddata-eval/README.md | 699 ++---------------- .../src/gooddata_eval/cli/main.py | 20 + .../core/agentic/_failed_runs.py | 44 +- .../core/agentic/_failure_message.py | 142 ++++ .../gooddata_eval/core/agentic/_outcome.py | 141 +++- .../core/agentic/_trace_linker.py | 5 +- .../gooddata_eval/core/agentic/alert_skill.py | 29 +- .../core/agentic/anomaly_detection.py | 22 +- .../core/agentic/conversation.py | 12 +- .../core/agentic/dashboard_skill.py | 31 +- .../core/agentic/document_skill.py | 9 +- .../core/agentic/general_question.py | 17 +- .../gooddata_eval/core/agentic/guardrail.py | 17 +- .../gooddata_eval/core/agentic/kda_skill.py | 22 +- .../core/agentic/metric_skill.py | 39 +- .../core/agentic/visualization.py | 133 +++- .../src/gooddata_eval/core/agentic/what_if.py | 22 +- .../src/gooddata_eval/core/config.py | 11 + .../core/evaluators/visualization.py | 50 +- .../src/gooddata_eval/core/models.py | 30 +- .../src/gooddata_eval/core/outcome.py | 143 ++++ .../src/gooddata_eval/core/scoring.py | 90 ++- packages/gooddata-eval/tests/conftest.py | 1 + .../tests/test_agentic_alert_skill.py | 20 +- .../tests/test_agentic_anomaly_detection.py | 15 +- .../tests/test_agentic_conversation.py | 28 +- .../tests/test_agentic_dashboard_skill.py | 18 +- .../tests/test_agentic_failure_message.py | 356 +++++++++ .../tests/test_agentic_kda_skill.py | 30 +- .../tests/test_agentic_metric_skill.py | 80 +- .../tests/test_agentic_observe_experiment.py | 12 + .../tests/test_agentic_visualization.py | 34 +- .../tests/test_agentic_what_if.py | 15 +- packages/gooddata-eval/tests/test_cli.py | 58 +- .../tests/test_context_date_filter.py | 400 ++++++++++ .../tests/test_eval_error_scores.py | 162 ++++ packages/gooddata-eval/tests/test_outcome.py | 441 +++++++++++ 38 files changed, 2685 insertions(+), 715 deletions(-) create mode 100644 packages/gooddata-eval/src/gooddata_eval/core/agentic/_failure_message.py create mode 100644 packages/gooddata-eval/src/gooddata_eval/core/outcome.py create mode 100644 packages/gooddata-eval/tests/test_agentic_failure_message.py create mode 100644 packages/gooddata-eval/tests/test_context_date_filter.py create mode 100644 packages/gooddata-eval/tests/test_eval_error_scores.py create mode 100644 packages/gooddata-eval/tests/test_outcome.py diff --git a/packages/gooddata-eval/AGENTS.md b/packages/gooddata-eval/AGENTS.md index 183cb387c..c380c1ae9 100644 --- a/packages/gooddata-eval/AGENTS.md +++ b/packages/gooddata-eval/AGENTS.md @@ -34,6 +34,7 @@ experiment. The newest and most actively developed package in the repo. | `core/reporting/` | console and JSON output rendering | | `core/scoring.py`, `core/runner.py` | scoring and orchestration | | `core/models.py` | `DatasetItem`, `ChatResult`, `ItemReport` and friends | +| `core/outcome.py` | `Outcome` (PASS / FAIL / ERROR): the one fail-closed rule that labels a failing run ERROR when the harness failed (simulated user, judge), plus the item label and the recorded-only shadow verdict. Labels decide no verdict; the agentic kinds reach it through `core/agentic/_outcome.py`'s `label_runs` | **Depends on**: `gooddata-sdk`, `httpx`, `pydantic`, `orjson`, `rich`. The LLM-judge evaluator is an optional extra (`llm-judge`, pulling `openai>=1.45,<2.0`); every `openai` @@ -161,6 +162,7 @@ Run this before guessing a `--model` string. | `GOODDATA_EVAL_TRACE_LABELS` | `key=value,...` labels stamped on every observation of a chat conversation via W3C baggage; `model_version` also sets the observation `version`, so set it only for a one-model run: every `--model` in one process gets the same labels. Keys must match `[A-Za-z0-9_-]+`; others are skipped with a warning. Dashboard-summary and document-takeaway runs are not labelled | | `GOODDATA_EVAL_JOIN_GENAI_TRACE` | on unless set to `0`, `false`, `no`, `off` or empty. An agentic item's chat turns ask gen-ai to open them under gd-eval's experiment root, so each item is one Langfuse trace with its scores written once. A gen-ai that drops caller baggage keeps today's separate traces; see the gotcha below | | `GD_EVAL_TIMERS` | same as `--timers` | +| `GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER` | same as `--accept-context-date-filter`; off by default. Visualization scoring accepts a date filter equal to one of the dashboard's in the item's `user_context` when the expected output has none (`check_filters`, see the README) | A gitignored `.env` at the repo root is the normal place for these; load it with `set -a && . ./.env && set +a` before the command. diff --git a/packages/gooddata-eval/README.md b/packages/gooddata-eval/README.md index dfda520f4..9f4133d4a 100644 --- a/packages/gooddata-eval/README.md +++ b/packages/gooddata-eval/README.md @@ -133,6 +133,7 @@ gd-eval run \ | `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. | | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. | | `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail`, `dashboard_summary` and `document_takeaway`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. | +| `--accept-context-date-filter` | off | Visualization scoring also accepts a date filter the agent carried over from the dashboard in the item's `user_context` when the expected output has none. Changes verdicts, so it is opt-in; also settable via `GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER=1`. See *Date filter carried over from the dashboard* below. | | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` and `document_takeaway` items go through their own endpoints, which have no such option. | **Concurrency and workspace safety.** Agentic kinds that create workspace objects @@ -265,642 +266,86 @@ the same reason. When some runs were ungraded the note says so instead of callin `agentic_conversation` has no K gate, so `--gate power` is refused for a dataset containing one rather than labelling a report `power` that only part of the dataset was decided under. -Each item additionally carries a per-phase breakdown: +**Which runs failed, and why.** A failure message names every failing run right after the gate note, with the +reason its loop stopped and only the checks it failed (expected/actual where the kind knows them), then the best +run's summary as before: -```json -"latency_breakdown_s": { - "agent_s": 4.02, // GoodData's own response time — the system under test - "judge_s": 1.31, // LLM-as-judge scoring, post-hoc - "simulated_user_s": 0.0, // our simulated user composing the next turn (multi-turn kinds) - "langfuse_s": 5.70 // trace lookup + score writing, off the critical path -} ``` - -Every item reports `runs_ungraded` beside `runs_passed`: runs the agent answered but the LLM judge returned -nothing readable for. Agentic items also list them as `unscored_runs` / `judge_errors` in their `detail` (a document item whose takeaway-endpoint request failed -lists that failure under `request_errors` instead), and a -`dashboard_summary` item carries `ungraded_criteria`. Such a run — or, for `dashboard_summary`, such a -criterion — is excluded from pass@K and from the quality score rather than counted as a failure: scoring it 0 -would be indistinguishable from the judge genuinely failing the answer, which is the confusion -`JudgeResponseError` exists to end. `pass@K` still holds on the runs that *were* graded, so an item can pass with -`runs_ungraded` set; `pass^K` cannot, because a run nobody graded leaves "all K passed" unverified. For -`dashboard_summary` an ungraded `must_include` / `must_not_include` criterion likewise cannot carry a pass — -"the judge could not tell" is not evidence the fact is present — while an ungraded `rubric` line only narrows -the quality score. When *no* run could be graded (for `dashboard_summary`: no gating criterion on any run) the -item errors instead of reporting failures. A non-zero count means pass@K was computed over fewer runs than -`--runs` asked for, so treat the result as weaker evidence and check the judge (`GD_EVAL_JUDGE_DIAGNOSTICS=1`, -or raise `JUDGE_MAX_COMPLETION_TOKENS` if the cause is `finish_reason=length`). The console says so in `Notes` -(`1 run(s) ungraded`, `2 criterion(s) ungraded`). - -`agent_s` + `judge_s` + `simulated_user_s` are the instrumented parts of the item's `latency_s`; they do not add -up to it exactly, because `latency_s` is wall-clock around the whole item and also covers the conversation -create/delete round trips, SDK construction and any cleanup. `langfuse_s` sits **beside** `latency_s`, never -inside it, because trace linking runs outside every item's critical path (see above) — summing all four would -re-inflate exactly what that design removes. - -A phase that a kind does not have reports `0.0` rather than an invented number, so read the zeroes as "not -applicable here", not "instant". Today: - -| Field | Populated by | -|---|---| -| `agent_s` | `agentic_general_question`, `agentic_metric_skill` | -| `judge_s` | `agentic_general_question` only — `agentic_metric_skill` compares MAQL by string, it has no LLM judge | -| `simulated_user_s` | `agentic_metric_skill` only — `agentic_general_question` is single-turn, it has no simulated user | -| `langfuse_s` | every agentic kind, but only on the `gd-eval` path and only when Langfuse credentials are present | - -The other six agentic kinds report `0.0` for the first three. Trace linking itself happens whenever -`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported, with or without `--langfuse`, because each -`evaluate_agentic_*` falls back to `try_make_langfuse_client()`. But its *duration* is measured by the CLI -runner rather than by `evaluate_agentic_*`, so a direct library caller sees `langfuse_s: 0.0` even though its -linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogether. - -### Latency: single-shot and multi-turn - -`avg_latency_s` is one number over a mixed population. The report instead keeps two kinds of item apart, by -their eval kind, and states latency for each: - -- **Single-shot** — kinds that send exactly one chat message per run: every non-agentic kind, plus - `agentic_general_question`, `agentic_guardrail` and `agentic_search`. -- **Multi-turn** — kinds that drive a simulated user or fixture turns: `agentic_visualization` (and - `vis_agentic`), `agentic_metric_skill`, `agentic_alert_skill`, `agentic_dashboard_skill`, - `agentic_document_skill`, `agentic_kda_skill`, `agentic_anomaly_detection`, `agentic_what_if` and - `agentic_conversation`. - -```json -"summary": { - "by_mode": { - "single_shot": {"total": 12, "passed": 10, - "traces_s": {"n": 24, "mean": 5.39, "trimmed_mean": 5.36, "min": 3.88, "p50": 5.33, "p70": 6.23, "p90": 6.82, "p99": 7.18, "max": 7.22}}, - "multi_turn": {"total": 6, "passed": 4, - "traces_s": {"n": 45, "mean": 6.91, "trimmed_mean": 6.84, "min": 2.78, "p50": 6.52, "p70": 8.03, "p90": 10.79, "p99": 13.57, "max": 14.25}, - "runs_s": {"n": 15, "mean": 23.55, "trimmed_mean": 23.31, "min": 12.64, "p50": 21.23, "p70": 28.52, "p90": 33.02, "p99": 36.94, "max": 37.51}} - } -} +Gate pass^3 failed: 2/3 runs passed — unstable, not a clean failure. Failing runs: run 2/3 FAIL (loop exit: success): filters_correct[date: expected none, actual ...]. ``` -`by_mode` is in each run's `summary` and in its `comparison` entry. `total` and `passed` count the evaluated -items of that mode (skipped ones belong to neither). Every distribution is `null` when nothing was timed. -Each item also states its `mode`, its `trace_latencies_s` and its `run_latencies_s`. - -- **Per trace is the main figure.** gen-ai starts one trace per chat message, so a trace is one chat turn. - Its latency is that turn's response time, from sending the message to the end of its stream - (`turn_wall_clock_sec`), recorded where the chat client completes the turn. It holds none of the LLM judge, - the simulated user or conversation create/delete. `dashboard_summary` is timed the same way around its one - request. A turn cut off by its turn or item deadline, or one that ended without a final answer (the agent's - iteration or token limit, a content filter), still counts, with the time it ran for: those are the slowest - turns, and the run that scored them is in the per-run figures too. A turn that ended in any other error (an - error event, or a 503 that outlasted its retries) does not: the time is that of a failed request, not a - response, and would only add a fast outlier. On the single-shot path such a turn ends its item as an error, so - it has no per-run figure either. An item that errored outside its evaluator's own failure reporting - contributes neither. -- **Single-shot** runs are exactly one trace, so `traces_s` is also the latency of a run. -- **Multi-turn** runs span several traces, so they get both views: `traces_s` over every turn of every - conversation (comparable with single-shot), and `runs_s`, the wall clock of each whole conversation from just - before its first message to the end of its scoring — every turn, simulated-user reply and judge call, but not - conversation create/delete. `agentic_conversation` drives its fixture once and contributes one run per item. - The two single-shot and multi-turn populations are never mixed. -- **Percentiles** use linear interpolation between closest ranks (NumPy's default), so `p50` is the median and - the report carries no separate median field. On a small sample the high percentiles sit at or near the - maximum: P99 over fewer than 100 samples is effectively the slowest one, which is why `n` always travels with it. -- **Trimmed mean** (`trimmed_mean`) is the average of the middle 80%: the fastest 10% and the slowest 10% of the - samples are dropped from each end (`n * 10 // 100` of them, so 25 samples drop two each) before averaging. It is - not the median, which is one middle value (`p50`). It is `null` below 10 samples, where there is nothing to - trim and the figure would only repeat `mean`. -- **Older reports** have no `by_mode`. The HTML report says so and nothing is backfilled. - -`avg_latency_s` and the other existing fields are unchanged. The console prints each run's pass counts per -mode and one `Latency, (n=...)` line per distribution beneath its summary line, and Trace -P50/P90/P99 columns (single-shot) in the multi-model comparison. +The same data is in each `failed_runs[]` record: `exit_reason` (the run's `LoopExit`, `null` for a kind without +one), `outcome` (see below) and, for the visualization, alert, metric and dashboard skills, `failed_checks` +(`[{"name", "label"?, "expected"?, "actual"?}]`, each `expected`/`actual` longer than 160 characters stored cut to +that length; the run's `detail` keeps the whole value). ---- +None of this changes a verdict, a score or an exception type; it only says more about each failure. -## `gd-eval report` +### Harness errors are labelled, not rescored -Turns JSON report(s) into one HTML file you can actually navigate. No server, no -credentials, no external assets — it opens over `file://`, attaches to a Jira issue and -survives a Slack thread. It follows GoodData's product look (colours, type, status chips). The faces are Inter, with -Funnel Display for headings, named but never loaded, so a viewer without them gets the system sans and the file stays -self-contained. +A run can fail because the test harness failed, not the agent: the simulated user's own model call fails, or the +judge cannot grade the answer. Each failing run is labelled in `failed_runs[].outcome` +(`{"status": "fail" | "error", "reason": ...}`) by one rule (`core/outcome.py`, `classify_run`). The rule is +fail-closed -- only these causes are `error`, anything else, an unknown cause included, is `fail`: -```bash -# one run -gd-eval report results.json -o report.html - -# several runs side by side -- each file becomes its own column -gd-eval report aug-21.json sep-07.json -o comparison.html --title "H200 regression check" - -# customer-safe -gd-eval report results.json -o customer.html --redact -``` - -| Flag | Description | +| Cause | Label | |---|---| -| `-o, --out PATH` | Where to write the HTML. Required. | -| `--title TEXT` | Title shown in the report header. | -| `--redact` | Drop conversation/response ids and raw reasoning, and rename models to `Model A`, `Model B`, … A model keeps its letter across all its runs, which are then numbered (`Run 1 · Model A`, `Run 2 · Model A`). Pass rate, per-item results, questions and latency survive. | - -The report is a *view* over the JSON. Unfiltered, every figure on it is read from the report, so it cannot -disagree with the console or the JSON. Narrow the selection with a filter and the figures are recomputed in the -page from the selected items' own records, by the same rules gd-eval uses. The page reads top to bottom as a -document, in plain words: a *test case* is one prompt given to the assistant, an *attempt* is one go at it, and the -two formats are *single-question* tests (one answer, no follow-up) and *conversation* tests (a back-and-forth with a -simulated user), which are `single_shot` and `multi_turn` in the JSON. - -- **A masthead** with the report's title (`--title`, or "Run history", "Run comparison", "Model comparison" or - "Run results" plus the model and workspace), when it was generated in the reader's own time zone, and a **Run setup** tooltip listing what - is the same in every run (model, provider, workspace, dataset, test cases, attempts per test case, pass rule) and - what differs. Results from different setups are not directly comparable, so a caution icon on a latency row and the - *New* / *Removed* marks in the test case table say where two runs stopped being alike. -- **View by** — a dropdown above the filters that lays the same report out two ways. **Date** is the layout the - rest of this list describes: a column per run, oldest to latest by start time, with change lines, sparklines and the latency line - chart. **Model** compares language models instead: a column per model, ranked best first by the console's winner - rule (pass rate, then quality, then the quicker average), with no time axis. Where a model has several runs its - latest stands for it, so each model is read at its most recent state. The Run filter becomes a Model filter, the hint - beside the dropdown says what is being compared (and names the dates when the compared runs differ in date), and the - Run setup tooltip does not flag the model itself as a difference. Every report opens on Date, including one that - compares several models (`gd-eval run --model a --model b --html`); choose Model to compare them. A single run has - nothing to switch between, so the dropdown is hidden. In the Model view the page reads, top to bottom: - - **Results** — one row per model, best pass rate first, with a bar of how its evaluated test cases ended (passed - green, failed red, could not be graded orange), so the green part is the pass rate beside it, and the **Median - latency** of one attempt (half the attempts were faster, half slower). A list under the table says what each term means. The counts and the - caution icon (a model evaluated on different test cases than the one with the most) are the date view's Outcomes - table. There are no headline cards by model: the table is the first thing on the page. - - **Quality against speed** — each model as a point: the share of test cases it passed against the median time of - one attempt, the same pass rate as the Results table and the same median as the latency tables. Up and to the left is better. Time runs from zero; - the pass rate uses the range the models span. It needs two models with timings. - - **By category** — a heat map of the pass rate per category and model, the best model in each row marked, the - categories where the models differ most first. Every cell prints its share and its counts, so the shade is never the - only signal. This is where a model that wins overall but loses on one category shows. - - **Consistency across attempts** — per model, the share of test cases that passed on at least one attempt (pass@K, - the light bar) beside the share that passed on every attempt (pass^K, the dark bar), and how many passed only some. - A long gap is a model that can do the task but not reliably. It appears only for a report that recorded - `pass_power_k` and attempted a test case more than once. - - **Latency sections** — the same four percentiles, one row per model, quickest first, with a bar per figure drawn - from zero beside the numbers. The quickest and slowest model in each column are marked as in the date view. - - **Test cases** — one result column per model in ranked order, a response-time column per model, and an **Across - models** column in place of *Since previous run*: *Mixed* (some models passed and some did not; hover to see who), - *All pass* or *All fail*. A skipped or ungraded test case is left out of the verdict. The table opens on the - *Mixed* ones, the test cases where the models differ. -- **Filters in two tiers** — the **Filter the report** bar at the top (category, format, result and run) decides what - the whole page covers: the Results cards, the outcomes table, the latency tables and the test case table all follow it, so the runs stay - comparable. The Test cases section repeats those four and adds the search box and the expression; they narrow the - table only, inside whatever the top bar allows, and only offer what the top bar leaves. Category, result and run are - attribute filters like the product's: a search box, an All row, the values with an **Only** shortcut and a count, - then Cancel and Apply. Changes are staged in the popup and take effect on Apply; closing it any other way discards - them. Nothing ticked means all. The run filter chooses which runs are shown; a result matches a test case when any of - the runs in play has it (so "Failed" over previous and latest finds every test case that broke in either), and the - expression is evaluated once per run in play with the same any-run rule. A filtered page states how many test cases - it covers, and each tier has its own Reset. -- **Results** — three headline cards for the latest run: test cases passed (of those evaluated: skipped ones are not - counted, as in the console's percentage), the average quality score and the median time per attempt. That last one - pools every format on the page and counts whole attempts, grading included, so it matches neither per-response latency table - below it; its tooltip says which formats it covers. With an earlier run each card carries one change line in words, "down from 86% on Sep 28", rather - than a signed number. The line is green when that direction is good and red when it is not, with a chevron and - screen-reader text (*Better* / *Worse*) so colour is not the only signal. From three runs each card also has a - sparkline of every run: hover a point for its value, click it to compare the latest run with that one instead of the - previous. The dot being compared with is drawn white, and a key under the cards says so (latest run, compared with, - other runs) and that a click on any other dot moves the comparison; it shows only when a sparkline does. A pick that - a filter has removed, or that has become the latest run, falls back to the previous run. The - pass-rate card links to the test case table to see *which* ones changed; the split by format lives in that table's - filters. -- **Outcomes** — how every test case ended, as a table with one row per run, newest first, in the same layout as the - latency tables. A run is identified by three columns: the **Run date**, the day and time it started (`summary.started_at`, - shown in the reader's own time zone, the UTC stamp on hover; a report written before it existed shows the date its file is - named for, without a time), and the **Run name**, the name the run was given, as given, so a run named `2026-09-28` reads - `2026-09-28` here although the charts style it as "Sep 28", and the **Model** it evaluated, its provider on hover. Then **Passed**, **Failed**, **Could not be graded** (an error left it without a verdict) and - **Skipped** (not run), then the **Pass rate**. The four are the values of the Result filter, and the pass rate is - the card's: passed out of the evaluated test cases, skipped ones left out, as in the console. The highest pass rate - is marked green with an up chevron and the lowest red with a down chevron, and the latest run is marked in lime. A - caution icon flags a run evaluated over a different number of test cases than the one before, because its pass - rate is then not a like-for-like change. The counts are read from the run's `summary` (`passed`, `failed`, - `errored`, `skipped`); a filtered view recounts them from the selected test cases, and a report written before - `summary.failed` existed shows what the other three leave. -- **Latency sections** — **Single-question tests · response time** first, then **Conversation tests**, which shows - both per response (each assistant reply, one gen-ai trace) and per conversation (start to finish, with the - simulated user and the judge). Each opens with a **line chart** of the median (P50), P70, P90 and P99 across the - runs, oldest on the left and the latest on the right, drawn from zero so line heights compare honestly; the lines - share one blue that darkens towards the tail. Hover or focus a run for its crosshair and every figure, in a tooltip - that says how many responses it rests on. A line needs two runs with timings, so with one run the chart is left - out. End labels name each line and give its latest value, and are dropped when two would touch. Under the chart a - table has one row per run, newest first, identified by the same **Run date**, **Run name** and **Model** columns as the outcomes table, with the same four figures. The quickest run in each column is marked - green and the slowest red, each with a chevron (down for least time, up for most) so colour is not the only - signal, and the latest run is marked in lime. A caution icon flags a run timed on a different number of samples - than the one before. Unfiltered they are read from `by_mode`; a filtered view recomputes them in the page by the - same rules. The fastest and slowest single response, the average and the trimmed average stay in the JSON (P99 - over a few dozen samples sits close to the slowest one, which is why `n` travels with it in the tooltip and - the cell hint). Seconds read with one decimal and no redundant ".0" (`3s`, `4.6s`). -- **A test case table**, newest first: the latest run's result, a **Since previous run** column, then the earlier runs - going back in time (click any result to open the drawer on that run). Each run's column is headed by its run date over - its run name, one to a line (hover the heading for the full run label); by model the columns are headed by the model. The change column reads *Regressed* (passed in - the previous run, not in the latest), *Fixed*, *Flaky* (flipped between pass and fail more than once), *New* - (in no earlier run) or *Removed* (in an earlier run, gone from the latest). A test case skipped in the latest run - has no label, since the change would describe earlier runs. With several runs the table opens on the regressions. Two time columns - keep the populations apart: **Response time** is the median of the test case's assistant responses, and **Time per - attempt** is the mean wall-clock of a whole attempt (the responses plus the judge and, for conversations, the - simulated user). With more than two runs shown they follow the latest run only. Hover a value to see every sample - behind it; the drawer labels its two latency chips the same way. A report written before per-trace timing has no - response figure and shows `avg_latency_s` as the attempt figure. -- **An expression filter**, folded under *Advanced: filter by a check expression* in the Test cases section, for - cross-cutting questions the fixed filters can't anticipate, e.g. `d.filter_ranking_score === false` or - `d.expected_metric_uris.length > 1 && !d.metrics_correct`. Available variables: - `d` (the checks of the run being read, its `detail`), `it` (that run's result), `i` (the row, `i.per[label]` for any - run), `q` (question), `kind` (category id). -- **The conversation**, when the test case ran the agentic multi-turn path — every turn in - order, with the simulated user marked apart from a real question, so you can see - whether the agent got there or was handed the answer. -- **A per-test-case drawer** — checks as pass/fail chips, expected vs actual side by side, - the latency timeline, conversation/response ids. -- **A latency timeline** from `detail.latency_breakdown`, in execution order, one bar per - step. Every step has a chevron that opens its full record in place, joined by `index`: the - paragraph a reasoning step was summarised from, or a tool call's arguments and result from - `detail.tool_calls`. A reasoning paragraph's markdown `**title**` is shown as bold text, - not as asterisks. Any number of steps can be open at once, and the chevron above the list - opens or closes all of them. The reasoning therefore lives on the timeline; a separate - Reasoning section appears only for paragraphs no step points at (or when a run recorded no - timeline). -- **A print stylesheet** that drops the filters and the drawer, so "Save as PDF" gives a clean page. - -A report that merges several files names each run after its file, and a file named for its date (`2026-09-28.json`) -reads as "Sep 28". A model is read from the run itself (`model` and the provider), never from the file name, so files -named for their model work as well as files named for their date. In the Date view the page treats the runs as one -model followed over time; when a report holds several models it adds the model to each run's name and points at -**Model** (see above) for the comparison. - -`--redact` additionally drops `transcript` and `tool_calls`: the exchange shows that the -simulated user is primed with the expected output, and a tool result carries -semantic-layer internals and real query rows. The turn count and the timeline shape -survive. - -Passing several files keyed by file name is the whole run-over-run mechanism: no -database, no run registry, just the JSON files you already have on disk. - ---- - -## `gd-eval models` - -List all LLM providers and their models in the org. Marks the active model -for a workspace when `--workspace` is given: - -```bash -gd-eval models \ - --host https://your.gooddata.cloud \ - --workspace ecommerce_demo -``` - -``` -┃ Provider ┃ Provider ID ┃ Model ID ┃ Family ┃ Active ┃ -│ Foundry4o │ foundry_… │ gpt-5.2 │ OPENAI │ ◀ active │ -│ │ │ gpt-4o │ OPENAI │ │ -│ HN_Anthropic │ hn_anthr_… │ claude-opus-4-7 │ ANTHROPIC │ │ -``` - ---- - -## `gd-eval generate` - -Reverse-engineers a `visualization` dataset out of the charts a customer has already -built, so you get eval questions without hand-authoring any. Reads the workspace's -declarative analytics model (read-only), translates each visible insight's buckets, -sorts and filters into an `expected_output.visualization` spec, then asks an LLM to -write the analyst question that chart answers. Because `expected_output` is copied from -a live object rather than authored, every question is grounded in the real LDM by -construction — the LLM only writes English. - -**Setup:** host + token (read access to the workspace), and `OPENAI_API_KEY` plus the -`llm-judge` extra for the phrasing step (`uv add 'gooddata-eval[llm-judge]'`; skip both -with `--no-phrase`). - -```bash -export GOODDATA_TOKEN='your-api-token' - -# 1. see what a workspace yields before writing anything -gd-eval generate \ - --host https://your.gooddata.cloud \ - --workspace ecommerce_demo \ - --dataset-name ecommerce \ - --dry-run - -# 2. generate, phrase, validate, and export -gd-eval generate \ - --host https://your.gooddata.cloud \ - --workspace ecommerce_demo \ - --dataset-name ecommerce \ - --dashboard dash_1_returns \ - --out ./my-dataset \ - --langfuse-out out/langfuse-dataset.json - -# 3. run it -gd-eval run --host … --workspace ecommerce_demo --dataset ./my-dataset --model gpt-5.2 -``` - -`--workspace` is where insights are read from; `--dataset-name` is the `dataset_name` -written into every item (and the default output folder). - -| Flag | Effect | -|---|---| -| `--dashboard ` | restrict to insights on that dashboard (repeatable); default is the whole workspace | -| `--out ` | output folder (default `./`); this is what `gd-eval run --dataset` reads | -| `--snapshot-out` / `--snapshot-in` | save/replay the fetched model — replay needs no host, token, or network | -| `--langfuse-out ` | also write a Langfuse-importable dataset JSON | -| `--id-prefix` | prefix exported Langfuse item ids (they're unique per *project*, so re-importing an item under its original id is a 409) | -| `--no-phrase` | skip the LLM; emit mechanical `Show ` questions | -| `--phrase-model` | OpenAI model for phrasing (default `gpt-4o`) | -| `--no-viz-type` | always blank the expected chart type | -| `--enrich-ranked <N>` | additionally derive up to N ranked questions (see below); default 0 (off) | -| `--skip-ambiguous` | drop items naming something the model carries more than once; reported either way | -| `--min-questions` / `--min-shapes` / `--min-filtered` | quality gate, default 15, 3 and 1 | - -### Ranked questions (`--enrich-ranked`) - -Analysts sort in Analytical Designer and save the chart without persisting the sort, so -`sort_by`/`ranking_filter` coverage is near zero on most real models — the eval can -punish a spurious ranking but never confirm the agent builds a required one. -`--enrich-ranked N` fills that gap by *deriving* ranked items from the specs already -extracted. Adding a limit or a sort to a definition that executes cannot make it -unanswerable, and "the top 3 X by Y" has exactly one correct spec, so a derived item is -less ambiguous to grade than the insight it came from. - -The budget is spent best-grounded first: - -1. **Insights whose own title promised a ranking their definition never implemented** — - "Top Returned Reasons" saved with `sorts: []`. The direction comes from the title - (`highest`/`most`/`largest` vs `lowest`/`least`/`worst`) and the N too when it states - one; a title naming both ends names neither and is still skipped. -2. **Ranking filters added to a plain breakdown** — one metric, one non-date dimension, - no existing sort. N follows the dimension's element count, so a top-5 over six values - is never emitted. -3. **Sort-only variants**, which order without limiting. - -Eligibility is deliberately narrow: two metrics leave "top 3 by what?" unanswered, a -second dimension leaves the N ambiguous between the pair and within a group, and a date -dimension turns the result into "top 3 months", which nobody asks. Variants are -deduplicated by resolved definition — differently-titled insights over one metric and -dimension would otherwise produce the same question twice — and bases are taken -round-robin by metric so one popular metric cannot become a third of the corpus. - -Derived items carry `derived_from` (the insight id) and `derived_basis` (`title` when a -human's chart title asked for the ranking, `shape` when this generator chose to add -one), so a pass rate over each can be computed separately. - -### Items that cannot say what they mean - -Two classes of question are unwinnable however well the agent behaves, and both are -reported: - -- **A name the model carries more than once.** One workspace has six labels all titled - "Product Title"; a question naming one cannot say which is meant, and a perfect chart - over the wrong one scores zero. `--skip-ambiguous` drops them; the count and the - offending names are printed either way. -- **A date granularity's cyclical twin.** `MONTH` walks consecutive calendar months, - `MONTH_OF_YEAR` stacks every January together. Date dimensions are therefore briefed - by what they do ("one point per calendar month over time, not month-of-year") and the - writer is told to say it in natural words while keeping the date dataset's name — - never as a label id in prose ("Order Created At - Month"). - -**The question must never contradict its own expected output.** Four rules enforce that: - -- The writer is briefed on buckets, sorts and filters only — never the insight title, - and never the chart type. Titles routinely describe intent the definition doesn't - implement ("Products by Most Items Sold" over `sorts: []`). -- Every generated question is checked against its spec, and any hit is a hard error: - ranking words (`top`, `most`, `highest`, …) require a real sort or ranking filter; - filter words (`only`, `last quarter`, `in 2025`, …) require a real date or attribute - filter; a breakdown clause requires a non-empty `view_by`/`segment_by` and vice versa; - a metric may never be broken down by itself; and no template residue (`breakdown - dimension`, `{…}`) may survive. A violation is fed back once for a rewrite, then - dropped — and a drop fails the run. -- The writer's rules are built per insight, so an insight with no `view_by` is never - asked to name a breakdown at all. -- `type` is set only when the question actually names a chart form. An insight's - `visualizationUrl` records what a human clicked, not what the question constrains — - with one exception: a chart with no breakdown *must* name its form ("as a KPI", "as a - single number"). Without it the agent reads a bare "Show me Gross Revenue" as a metric - lookup, activates only its search skill and builds nothing. - -Everything the writer sees is a display name (`Spend Amount`, `Merchant Name`), never a -raw URI, so questions read like a person wrote them. - -**What it won't do.** Insights it can't express without guessing are skipped with a -printed reason, never approximated: derived (arithmetic/PoP) measures, measure-level -filters, `uris`-form attribute filters, unmapped chart types, hidden objects, and -insights whose title promises behaviour their definition lacks (though `--enrich-ranked` -implements a promised *ranking* rather than discarding it). If too few survive, the -quality gate fails the run rather than fabricating items to hit the minimum — point at -more dashboards, or lower `--min-questions`. - -Every written item is validated as a `DatasetItem` with a scorable AAC visualization -before the command reports success. - ---- - -## Dataset format - -A dataset is a folder of `.json` files, one per question: - -```json -{ - "id": "stable-unique-id", - "dataset_name": "my_dataset", - "test_kind": "visualization", - "question": "Show revenue by quarter", - "expected_output": { } -} -``` - -Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`, -`search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`. - -### `visualization` items - -The `expected_output` lists the candidate charts in the AAC format, and the item passes when -the agent's chart matches one of them. Keys beside the candidates add item-level checks: - -```json -{ - "id": "sankey-001", - "dataset_name": "agent_visualization_extended_charts", - "test_kind": "visualization", - "question": "Show how net sales flow from product category to customer country.", - "expected_output": { - "visualization": { - "type": "sankey_chart", - "query": {"fields": {"m": {"using": "metric/net_sales"}, "c": {"using": "label/product_category"}, - "d": {"using": "label/customer_country"}}}, - "metrics": ["m"], - "from": "c", - "to": "d", - "buckets_strict": true - }, - "requires_visualization_part": true - } -} -``` - -Item-level keys, all optional except the candidates: - -- `visualization` — one candidate chart, or a list of them. `expected_outputs` - (`[{"visualization": {...}}, ...]`) is the same list in the agentic fixture form. -- `requires_execution` — the agent must also run the chart it built. -- `expects_visualization: false` — the agent must answer without a chart, for example by asking - for a part the request leaves out. The item then lists no candidates. -- `visualization_optional: true` — either an answer without a chart, or a chart that matches a - candidate, is right. -- `answer_must_include` / `answer_must_not_match` — regular expressions, matched without regard - to case against the final answer. Every `answer_must_include` pattern must match, and no - `answer_must_not_match` pattern may. -- `format_intents` — per metric uri, how the metric's numbers must read, for example - `{"metric/net_sales": {"divide_by": 1000000, "decimals": 1, "suffix": "M"}}`. -- `requires_visualization_part` — the chart must come back as a part of the answer, not only as - the arguments of the chart tool call. - -Switches on a candidate chart: - -- `buckets_strict` — compare bucket by bucket, not only the union of the buckets, so swapped - heatmap rows and columns or a reversed sankey fail. -- `metrics_ordered` — compare the metrics by position, for bullet roles and bubble axes. - -Any other key, at item level or on a candidate chart, makes the item error before any request -is sent, because a misspelled key would otherwise be dropped and its check would never run: - -``` -ValueError: the expectation states ['answer_must_inlcude'], which is not one of ['answer_must_include', ...] -ValueError: a candidate chart states ['bucket_strict'], which the chart model does not read -``` - -### `dashboard_summary` items - -Summary items call the dedicated summary endpoint -(`POST /api/v1/ai/workspaces/{ws}/summary`) instead of the chat endpoint, so -they carry an extra `summary_input` block, and the `expected_output` is a -**rubric** rather than an exact answer (summaries are free text): - -```json -{ - "id": "summary-001", - "dataset_name": "summary_pilot", - "test_kind": "dashboard_summary", - "question": "Summarize the Sales Overview dashboard.", - "summary_input": { - "dashboard_id": "sales_overview" - }, - "expected_output": { - "must_include": ["States the overall revenue trend", "Identifies the top segment"], - "must_not_include": ["Numbers or segments not present in the visualizations"], - "rubric": ["Reads as a coherent business summary"] - } -} -``` - -`summary_input` requires only `dashboard_id` (the endpoint summarizes the whole -dashboard). Optional fields narrow the scope: `visualizations` (list of ids), -`filter_context` (AFM filters), `tab_id`, and `format_hint`. - -The `expected_output` rubric: - -- `must_include` — facts a good summary must contain; **all** must pass for the item to pass. -- `must_not_include` — hallucination/accuracy guards; **any** violation fails the item. -- `rubric` — soft quality dimensions; they affect `quality_score` but do not gate pass/fail. - -Each criterion is scored independently by the LLM judge, so `quality_score` -is the fraction of satisfied criteria. - -### `document_takeaway` items - -Takeaway items call the AI Publisher takeaway endpoint -(`POST /api/v1/ai/workspaces/{ws}/takeaway`) with a `takeaway_input` block, authored in -snake_case and sent as the endpoint's camelCase body. In a Langfuse dataset the block goes -in the item's `metadata` (or its `input` object), and `input` can stay a plain case label; -an item carrying `takeaway_input` is a `document_takeaway` item unless it declares another -`test_kind`. The endpoint needs the `enableGenAiReportTakeaways` feature flag. - -```json -{ - "id": "takeaway-001", - "dataset_name": "agent_document_takeaway", - "test_kind": "document_takeaway", - "question": "P01 default format, Sales page, no instructions", - "takeaway_input": { - "visualizations": ["<visualization id>", {"visualization_id": "<id>", "filters": []}], - "filter_context": [], - "instructions": {"document": null, "slot": null, "format_hint": null}, - "max_characters": 300 - }, - "expected_output": { - "no_data": false, - "format": {"no_lists": false}, - "must_include": ["States one finding that connects at least two of the charts"], - "must_not_include": [], - "rubric": [] - } -} -``` - -`takeaway_input` requires `visualizations`; unknown keys are rejected. Every -`expected_output` key is optional: - -- `no_data` — whether the endpoint must answer `noData`; when either side says no data, - nothing else is graded. -- `format` — overrides of the deterministic checks, which gate pass/fail: `bold_headline`, - `max_headline_words` (10), `max_sentences` (2), `max_words` (50), `single_paragraph`, - `no_lists`, `no_headings`. `false` or `null` switches a check off; unknown keys are - rejected. `length` (`max_characters`), `no_ids` and `no_meta` always apply. -- `default_rules` — `false` drops the default judge criteria (one finding; no executive - summary, chart walk-through, background opener, figures, dates, causes the data cannot - show, chart description). -- `must_include`, `must_not_include`, `rubric` — the item's own judge criteria, graded like - `dashboard_summary`. - -`word_count` and `headline_word_count` are reported in the item detail. The default judge -criteria are calibrated with `make calibrate-takeaway` (needs `OPENAI_API_KEY`); re-run it -after changing their wording. - -The client retries 429/5xx answers and connection failures. A request error that survives -the retries is handled differently by the two entry points: `gd-eval run` aborts the item -with that error, whatever `--runs` is, while the CI entry point -(`evaluate_document_takeaway`) records it as an errored run, keeps the item's other runs, -and raises only when no run completed. A read timeout is not retried, and the CI entry -point records it as an errored run too. Any other 4xx and an unsupported URL scheme are -never retried and fail the item in both. - -### Items asked in a dashboard context (`user_context`) - -An item can be asked the way a user asks from an open dashboard or an attached widget. -Its `user_context` is sent verbatim as `userContext` on every chat message of the item, -follow-up and clarification messages included — the server treats a message without one -as a cleared context. Requires the `enableAiContextSetup` feature flag on the target -organization — without it the server ignores the dashboard view. Applies to chat items -only; `dashboard_summary` and `document_takeaway` items carry their scope in -`summary_input` / `takeaway_input` instead. No -`userContext` key is sent at all when neither the item nor, for `agentic_conversation`, -the current turn sets one. - -```json -{ - "id": "ctx-001", - "dataset_name": "my_dataset_ctx", - "test_kind": "general_question", - "question": "Which dashboard am I looking at, and what does it cover?", - "user_context": { - "view": { - "dashboard": { - "id": "sales_overview", - "title": "Sales Overview", - "widgets": [ - {"widgetType": "insight", "widgetId": "w1", "title": "Revenue by Month", "visualizationId": "revenue_by_month"} - ] - } - } - }, - "expected_output": "Names the Sales Overview dashboard and summarizes its charts." -} -``` - -The schema belongs to the AI chat API (`UserContext`: `view.dashboard`, -`referencedObjects`, `activeObject`), so it is not validated here beyond being an object. -In a Langfuse dataset, put it in the item `metadata` (or in the `input` object) under -`user_context`. An object in `input` wins and `metadata` is then not read; a missing or -`null` value in `input` falls back to `metadata`. The value used must be an object or `null`; -anything else fails the dataset load. +| `simulated_user_failed` | `error` | +| the judge or scorer could not grade the run (`judge_failed`) | `error` | +| `chat_error` (a gen-ai error event, a broken stream, a 5xx after its retries) | `fail` -- on the product path, a customer sees it too | +| `agent_timeout`, `turn_incomplete` (the agent was too slow, or ended its turn without an answer) | `fail` | +| `success`, `budget_exhausted`, `agent_silent`, `not_run`, anything else | `fail` | + +The label decides nothing: an `error` run counts as a run that did not pass, so every gate, pass rate and score is +what it was before labels existed. It shows up in four places: + +- the "Failing runs" part of the message: `run 1/3 ERROR (simulated_user_failed)`, without the run's checks, which + failed because the harness did; +- `failed_runs[].outcome`; +- on the raised exception, `outcome` = `{"status": "error", "reason": ...}` when **every** run of the item errored + (the item says nothing about the agent; its message starts `Harness error, no run was scored (<reason>).`), and + `shadow` -- what a candidate rule that leaves `error` runs out of the gate (at least ceil(K/2) scored runs, else + `error`) would have given: `{"verdict": "pass" | "fail" | "error", "runs_scored", "runs_total"}`. Recorded so the + rule can be judged on real runs; nothing uses it; +- in Langfuse, an `error` run gets an extra score `eval_error` = 1, commented with the reason. Its other scores are + written as always. + +A runner turns these into its own report. The gdc-nas Tavern harness, for example, adds them to the JUnit testcase +as properties and keeps the failure a failure. + +Applies to the agentic visualization, alert, metric, dashboard, KDA, anomaly detection and what-if kinds, and to +judge failures of the general question, guardrail and document skill kinds. `agentic_conversation` records its +turns' exits but does not label them yet. + +**Changed exit value:** a run whose turn timed out or ended without a final answer used to record `exit_reason` +`chat_error`; it now records `agent_timeout` or `turn_incomplete`, in the run detail and in `failed_runs[]`. Filters +on `exit_reason == "chat_error"` see fewer runs. The item budget (`--item-timeout`) runs from conversation creation, +so an `agent_timeout` from it can include the simulated user's replies and the chat retries' backoff. + +#### Date filter carried over from the dashboard (opt-in) + +A user who asks for a new chart from an open dashboard without naming a period may get +the dashboard's date filter on the chart, and the product accepts that. By default the +visualization scoring still requires the chart's date filters to equal the expected ones. +Pass `--accept-context-date-filter` to `gd-eval run`, set `GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER=1` +(`0`, `false`, `no`, `off` and empty mean off), or pass `accept_context_date_filter=True` to +`check_filters` to also accept, for the date filter only, a chart that: + +- has exactly one date filter, whose period equals a `date_filter` in + `user_context.view.dashboard.filters` (resolved against the same day), and +- belongs to an item whose expected output has no date filter. + +The dashboard's common date filter (no `using`) accepts any date dataset, one that names a +dataset accepts only that one; the one the chart used is reported in the run detail as +`filter_date_context: {"inherited": true, "dataset": ...}`, and a failure summary shows +`date : True (inherited from dashboard context)`. A chart with no date filter passes as +before; an expected date filter, a different period, ranking and attribute filters stay +strict. Applies to `visualization`, `vis_agentic` and `agentic_visualization` items; with the +variable unset the scores and the detail are exactly as without it. + +With the flag on, an expected output cannot say "all time, on purpose": an expected output +with no date filter then also accepts a chart that copied the dashboard's period. An item +that must reject the dashboard's period needs to run without the flag. `agentic_conversation` items take the item's `user_context` as the context of the first turn. A turn in `expected_output.turns` can set its own `user_context`, which applies diff --git a/packages/gooddata-eval/src/gooddata_eval/cli/main.py b/packages/gooddata-eval/src/gooddata_eval/cli/main.py index efbbf1069..579ac89fd 100644 --- a/packages/gooddata-eval/src/gooddata_eval/cli/main.py +++ b/packages/gooddata-eval/src/gooddata_eval/cli/main.py @@ -17,6 +17,7 @@ from gooddata_eval.cli.agentic_runner import AGENTIC_TEST_KINDS, UNGATED_AGENTIC_TEST_KINDS, run_agentic_items from gooddata_eval.core.chat.sse_client import ChatClient, set_default_item_timeout, set_default_turn_timeout from gooddata_eval.core.config import ( + ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, DEFAULT_GATE, DEFAULT_JUDGE_MODEL, JUDGE_MODEL_ENV_VAR, @@ -136,6 +137,15 @@ def _build_parser() -> argparse.ArgumentParser: "Off by default because a large run emits hundreds of lines; the same measurements " "are always in the JSON report's latency_breakdown_s. Equivalent to GD_EVAL_TIMERS=1.", ) + run.add_argument( + "--accept-context-date-filter", + dest="accept_context_date_filter", + action="store_true", + help="Visualization scoring also accepts a chart whose single date filter equals one of the " + "dashboard's in the item's user_context, when the expected output has no date filter. " + "Off by default because it changes verdicts. Equivalent to " + "GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER=1.", + ) run.add_argument( "--turn-timeout", dest="turn_timeout", @@ -309,6 +319,15 @@ def _apply_timer_flag(enabled: bool) -> None: os.environ[TIMERS_ENV_VAR] = "1" +def _apply_accept_context_date_filter_flag(enabled: bool) -> None: + """Publish --accept-context-date-filter for check_filters, like --timers. + + Only ever sets it -- never clears a value the caller exported themselves. + """ + if enabled: + os.environ[ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR] = "1" + + def _reject_power_gate_on_ungated_items(config: RunConfig, items: list) -> None: """Refuse a pass^K request the run cannot honour for every item. @@ -722,6 +741,7 @@ def sdk_factory(): def main(argv: list[str] | None = None) -> int: args = parse_args(argv if argv is not None else sys.argv[1:]) _apply_timer_flag(getattr(args, "timers", False)) + _apply_accept_context_date_filter_flag(getattr(args, "accept_context_date_filter", False)) _apply_judge_model(getattr(args, "judge_model", None)) if hasattr(args, "concurrency") and args.concurrency < 1: print("error: --concurrency must be >= 1.", file=sys.stderr) diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failed_runs.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failed_runs.py index 694928b9b..300121e08 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failed_runs.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failed_runs.py @@ -25,6 +25,8 @@ from collections.abc import Callable, Sequence from typing import Any +from gooddata_eval.core.agentic._failure_message import cap_failed_checks +from gooddata_eval.core.agentic._outcome import run_exit_reason, run_outcome from gooddata_eval.core.models import build_tool_calls @@ -33,6 +35,7 @@ def build_failed_runs( *, passed: Callable[[Any], bool], detail: Callable[[Any], dict], + failed_checks: Callable[[Any], list[dict]] | None = None, ) -> list[dict]: """One record per run that did not pass, in the order the runs happened. @@ -47,6 +50,14 @@ def build_failed_runs( A run the judge could not grade is recorded like any other non-passing run, with its ``error`` set. That matches ``core/runner.py``, which also records ungraded runs; the verdict-level accounting of ungraded runs stays with the caller's ``unscored_runs``. + + ``exit_reason`` is the run's ``LoopExit`` value, or null for a kind whose loop records + none, and ``outcome`` its ``core.outcome.Outcome`` record: ERROR when the harness failed + (its simulated user or its judge), FAIL otherwise. The label decides nothing; the run + counts as not passed either way. ``failed_checks`` is written only when the caller supplies it: a missing + key means the kind does not name its failing checks, while an empty list would claim the + run failed none. Its expected/actual values are capped in size; the run's ``detail`` + keeps them whole. """ records: list[dict] = [] for run_index, run in enumerate(run_results, start=1): @@ -54,19 +65,22 @@ def build_failed_runs( continue tool_calls = list(getattr(run, "tool_call_events", None) or []) reasoning_steps = list(getattr(run, "reasoning_steps", None) or []) - records.append( - { - "run_index": run_index, - "passed": False, - "error": getattr(run, "judge_error", None), - "detail": detail(run), - "conversation_id": getattr(run, "conversation_id", None), - "response_id": getattr(run, "response_id", None), - "reasoning_step_count": len(reasoning_steps), - "reasoning_steps": reasoning_steps, - "tool_call_count": len(tool_calls), - "tool_names": [getattr(e, "function_name", None) for e in tool_calls], - "tool_calls": build_tool_calls(tool_calls), - } - ) + record: dict[str, Any] = { + "run_index": run_index, + "passed": False, + "error": getattr(run, "judge_error", None), + "detail": detail(run), + "conversation_id": getattr(run, "conversation_id", None), + "response_id": getattr(run, "response_id", None), + "reasoning_step_count": len(reasoning_steps), + "reasoning_steps": reasoning_steps, + "tool_call_count": len(tool_calls), + "tool_names": [getattr(e, "function_name", None) for e in tool_calls], + "tool_calls": build_tool_calls(tool_calls), + "exit_reason": run_exit_reason(run), + "outcome": run_outcome(run, passed=False).as_record(), + } + if failed_checks is not None: + record["failed_checks"] = cap_failed_checks(failed_checks(run)) + records.append(record) return records diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failure_message.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failure_message.py new file mode 100644 index 000000000..b7c8ae902 --- /dev/null +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failure_message.py @@ -0,0 +1,142 @@ +# (C) 2026 GoodData Corporation +"""The head of an agentic kind's failure message: the gate result, then every failing run. + +Each kind's assertion message describes its BEST run, which on a partial gate failure (some +of the K runs passed) is a passing run: every check reads True and the reason the item +failed is nowhere in the message. This module renders the part that is missing -- for each +failing run its index, why its loop stopped and the checks it failed, with expected/actual +where the kind knows them -- from the same ``failed_runs`` records the item reports, so the +message and the triage data cannot disagree about which runs failed or why. + +One renderer for every kind on purpose: the readers of these messages (people and the +triage tooling) look for one shape, whatever the capability under test. +""" + +from __future__ import annotations + +import json +from collections.abc import Iterable, Mapping, Sequence +from dataclasses import dataclass +from enum import Enum +from typing import Any + +__all__ = ["FailedCheck", "cap_failed_checks", "failed_check_records", "failing_runs_note", "failure_head"] + +# A value longer than this is cut: the message is one line per run, and the whole value is in +# the run's detail anyway. +_MAX_VALUE_CHARS = 160 + + +class _Unset(Enum): + UNSET = "unset" + + +_UNSET = _Unset.UNSET + + +@dataclass(frozen=True) +class FailedCheck: + """One check a run failed, with the expected/actual pair when the kind knows it. + + ``label`` narrows the check when one name covers several parts, e.g. the ``date`` part of + ``filters_correct``. + """ + + name: str + expected: Any = _UNSET + actual: Any = _UNSET + label: str | None = None + + def as_record(self) -> dict[str, Any]: + """The JSON-ready form written into a failing run's ``failed_checks``.""" + record: dict[str, Any] = {"name": self.name} + if self.label is not None: + record["label"] = self.label + if self.expected is not _UNSET: + record["expected"] = self.expected + if self.actual is not _UNSET: + record["actual"] = self.actual + return record + + +def failed_check_records(checks: Iterable[FailedCheck]) -> list[dict[str, Any]]: + """``checks`` as the records ``failed_runs[].failed_checks`` holds.""" + return [check.as_record() for check in checks] + + +def _text(value: Any) -> str: + text = value if isinstance(value, str) else json.dumps(value, default=str, ensure_ascii=False, sort_keys=True) + return " ".join(text.split()) + + +def _value(value: Any) -> str: + if value is None or value == [] or value == {} or value == "": + return "none" + text = _text(value) + return text if len(text) <= _MAX_VALUE_CHARS else text[: _MAX_VALUE_CHARS - 3] + "..." + + +def cap_failed_checks(records: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]: + """``records`` with every expected/actual value cut to what the message shows of it. + + A check's values can be whole filter or bucket structures, repeated for every failing run + of every item; the record keeps the same ``_MAX_VALUE_CHARS`` the message does. A value + that fits stays as it is, JSON and all; a longer one becomes its cut text. The run's + ``detail`` holds the full values. + """ + capped: list[dict[str, Any]] = [] + for record in records: + entry = dict(record) + for key in ("expected", "actual"): + if key in entry and len(_text(entry[key])) > _MAX_VALUE_CHARS: + entry[key] = _value(entry[key]) + capped.append(entry) + return capped + + +def _render_check(record: Mapping[str, Any]) -> str: + name = str(record.get("name", "?")) + parts: list[str] = [] + if "expected" in record: + parts.append(f"expected {_value(record['expected'])}") + if "actual" in record: + parts.append(f"actual {_value(record['actual'])}") + label = record.get("label") + if not parts and label is None: + return name + inner = ", ".join(parts) + if label is not None: + inner = f"{label}: {inner}" if inner else str(label) + return f"{name}[{inner}]" + + +def _render_run(record: Mapping[str, Any], runs_total: int) -> str: + outcome = record.get("outcome") + head = f"run {record.get('run_index', '?')}/{runs_total}" + exit_reason = record.get("exit_reason") + if isinstance(outcome, Mapping) and outcome.get("status") == "error": + # A harness error: the reason (simulated_user_failed, judge_failed) is the diagnosis. + # Its checks are not listed: they failed because the harness did, not the agent. + return f"{head} ERROR ({outcome.get('reason') or 'unknown'})" + head += " FAIL" + (f" (loop exit: {exit_reason})" if exit_reason else "") + checks = record.get("failed_checks") + if checks: + return f"{head}: {', '.join(_render_check(c) for c in checks)}" + if checks is not None: + return f"{head}: no failing check recorded" + return head + + +def failing_runs_note(failed_runs: Sequence[Mapping[str, Any]], runs_total: int) -> str: + """Every failing run on one line each, joined with ``; ``. Empty when no run failed.""" + return "; ".join(_render_run(record, runs_total) for record in failed_runs) + + +def failure_head(gate_note: str, failed_runs: Sequence[Mapping[str, Any]], runs_total: int) -> str: + """``gate_note`` followed by the failing runs, ready to start a kind's assertion message. + + Returns ``gate_note`` unchanged when there is no failing run to list, so a kind that + builds no ``failed_runs`` keeps its message exactly as before. + """ + note = failing_runs_note(failed_runs, runs_total) + return f"{gate_note} Failing runs: {note}." if note else gate_note diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_outcome.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_outcome.py index 576aee838..f45c33290 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_outcome.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_outcome.py @@ -19,18 +19,147 @@ from __future__ import annotations +from collections.abc import Callable, Sequence +from dataclasses import dataclass +from enum import Enum from typing import Any, NoReturn +from gooddata_eval.core.agentic._failure_message import failure_head +from gooddata_eval.core.agentic._gate import gate_failure_note +from gooddata_eval.core.chat.sse_client import TurnIncompleteError, TurnTimeoutError from gooddata_eval.core.models import ( AgenticAssertionError, AgenticEvalOutcome, + LoopExit, ReasoningStepEvent, ToolCallEvent, timeline_detail, ) +from gooddata_eval.core.outcome import EVAL_ERROR_SCORE, Outcome, classify_run, item_outcome, shadow_verdict from gooddata_eval.core.timing import PhaseTimings -__all__ = ["agentic_detail", "agentic_success", "raise_agentic_failure"] +__all__ = [ + "CHAT_FAILURE_EXITS", + "EVAL_ERROR_SCORE", + "RunLabels", + "agentic_detail", + "agentic_success", + "attach_labels", + "chat_failure_exit", + "gate_head", + "label_runs", + "log_eval_error_score", + "raise_agentic_failure", + "run_exit_reason", + "run_outcome", +] + + +CHAT_FAILURE_EXITS = frozenset({LoopExit.CHAT_ERROR, LoopExit.AGENT_TIMEOUT, LoopExit.TURN_INCOMPLETE}) +"""Every exit ``chat_failure_exit`` can return: the loop stopped because a chat call raised.""" + + +def chat_failure_exit(exc: BaseException) -> LoopExit: + """The ``LoopExit`` for a run whose chat call raised ``exc``. + + ``TurnTimeoutError`` and ``TurnIncompleteError`` subclass ``ChatError`` but are the + agent's own failures -- too slow, or no final answer -- so they get exits of their own + instead of reading as a chat API fault. Anything else a kind catches there (a gen-ai + error event, a broken stream, a 5xx after its retries) is ``CHAT_ERROR``. All three are + FAIL in ``core/outcome.py``: a customer hits each of them too. + """ + if isinstance(exc, TurnTimeoutError): + return LoopExit.AGENT_TIMEOUT + if isinstance(exc, TurnIncompleteError): + return LoopExit.TURN_INCOMPLETE + return LoopExit.CHAT_ERROR + + +def run_exit_reason(run: Any) -> str | None: + """The run's ``exit_reason`` as a string, or None for a kind whose run records none.""" + value = getattr(run, "exit_reason", None) + if isinstance(value, Enum): + return str(value.value) + return value if isinstance(value, str) else None + + +def run_outcome(run: Any, *, passed: bool) -> Outcome: + """One run's label: ``classify_run`` over the run's exit and its judge error, if any.""" + judge_error = getattr(run, "judge_error", None) + return classify_run( + run_exit_reason(run), passed=passed, judge_error=judge_error if isinstance(judge_error, str) else None + ) + + +@dataclass(frozen=True) +class RunLabels: + """The label of every run of an item, in run order. + + The labels decide nothing: the kind gates on its own pass_at_k/pass_power_k, and an + ERROR run counts as a run that did not pass. They only say why a run did not pass, and + whether the whole item is a harness error (``item_outcome``). + """ + + outcomes: tuple[Outcome, ...] + + def errored_run(self, run_index: int) -> Outcome | None: + """The ERROR label of run ``run_index`` (0-based), or None when the run is not one.""" + outcome = self.outcomes[run_index] if 0 <= run_index < len(self.outcomes) else None + return outcome if outcome is not None and outcome.is_error else None + + @property + def item_outcome(self) -> Outcome | None: + """ERROR when every run errored, else None (see ``core.outcome.item_outcome``).""" + return item_outcome(self.outcomes) + + @property + def outcome_record(self) -> dict[str, Any] | None: + """``item_outcome`` as the record the exception carries, or None.""" + outcome = self.item_outcome + return outcome.as_record() if outcome is not None else None + + def shadow(self, gate: str | None) -> dict[str, Any]: + """The phase-2 candidate verdict, recorded on the exception and never used.""" + return shadow_verdict(self.outcomes, gate) + + +def label_runs(run_results: Sequence[Any], *, passed: Callable[[Any], bool]) -> RunLabels: + """The ``RunLabels`` of an item's runs; ``passed`` is the kind's own per-run verdict.""" + return RunLabels(tuple(run_outcome(r, passed=passed(r)) for r in run_results)) + + +def attach_labels(exc: BaseException, labels: RunLabels, gate: str | None) -> None: + """Set ``outcome`` (only when every run errored) and ``shadow`` on an exception a kind + raises by hand -- the counterpart of the two ``raise_agentic_failure`` arguments.""" + outcome = labels.outcome_record + if outcome is not None: + exc.outcome = outcome # ty: ignore[unresolved-attribute] + exc.shadow = labels.shadow(gate) # ty: ignore[unresolved-attribute] + + +def gate_head( + gate: str | None, labels: RunLabels, runs_passed: int, failed_runs: Sequence[dict], runs_ungraded: int = 0 +) -> str: + """The start of a kind's failure message: the gate result, then every failing run. + + An item whose every run errored says so first, so a reader sees at once that the agent + was never scored. The gate note itself is the one the item was judged by. + """ + runs_total = len(labels.outcomes) + note = gate_failure_note(gate, runs_passed, runs_total, runs_ungraded) + item = labels.item_outcome + if item is not None: + note = f"Harness error, no run was scored ({item.reason}). {note}" + return failure_head(note, failed_runs, runs_total) + + +def log_eval_error_score(ctx: Any, trace_id: Any, outcome: Outcome) -> None: + """Mark one ERROR run in Langfuse: ``eval_error`` = 1, its comment the reason. + + An extra score only. The run's pass, gate, check and quality scores are written as + always, so nothing read from Langfuse changes; the marker says why the run failed. + """ + ctx.score(trace_id, name=EVAL_ERROR_SCORE, value=1, data_type="NUMERIC", comment=outcome.reason) def agentic_detail( @@ -60,6 +189,8 @@ def raise_agentic_failure( timings: PhaseTimings | None = None, failed_runs: list[dict] | None = None, run_latencies_s: list[float] | None = None, + outcome: dict[str, Any] | None = None, + shadow: dict[str, Any] | None = None, ) -> NoReturn: """Build ``exception_cls(message)`` with every common field attached, and raise it. @@ -68,6 +199,10 @@ def raise_agentic_failure( "no verdict at all" branch raises a bare ``JudgeResponseError`` instead (not a subclass) does not go through this helper for that branch. + ``outcome`` is set only when every run errored (``RunLabels.outcome_record``), and + ``shadow`` is the phase-2 candidate verdict (``RunLabels.shadow``); both are labels a + runner reads, neither changes the failure. + ``timings`` stays optional: only the three kinds that track per-run ``PhaseTimings`` (general_question, metric_skill, dashboard_skill) pass one -- the rest keep their own ``PhaseTimings()`` zero default, same as before this helper existed. ``failed_runs`` @@ -88,6 +223,10 @@ def raise_agentic_failure( exc.failed_runs = failed_runs if run_latencies_s is not None: exc.run_latencies_s = run_latencies_s + if outcome is not None: + exc.outcome = outcome + if shadow is not None: + exc.shadow = shadow raise exc diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_trace_linker.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_trace_linker.py index 3f9d3028e..a549f6fef 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_trace_linker.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_trace_linker.py @@ -217,9 +217,10 @@ def export_unscored(self, run_idx: int, conversation_id: str) -> None: with self.observe(self.trace(conversation_id), run_idx, conversation_id=conversation_id): pass - def score(self, trace_id: Any, *, name: str, value: Any, data_type: str) -> None: + def score(self, trace_id: Any, *, name: str, value: Any, data_type: str, comment: str | None = None) -> None: """Write one score, swallowing Langfuse failures the way ``score_safe`` always has.""" - self._lf.score_safe(self._client, trace_id, name=name, value=value, data_type=data_type) + extra = {"comment": comment} if comment is not None else {} + self._lf.score_safe(self._client, trace_id, name=name, value=value, data_type=data_type, **extra) def quality(self, trace_id: Any, *, strict_checks: dict, latency_sec: Any, cost_usd: Any) -> None: """Write the derived quality/value scores for one run.""" diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py index f1f532af3..ee6a439cd 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py @@ -15,15 +15,23 @@ from gooddata_eval.core.agentic._catalog import AnomalyDetectionGranularity, CatalogMetricAlert from gooddata_eval.core.agentic._failed_runs import build_failed_runs +from gooddata_eval.core.agentic._failure_message import FailedCheck, failed_check_records from gooddata_eval.core.agentic._gate import ( DEFAULT_GATE, EvalGate, - gate_failure_note, gate_passed, log_gate_scores, stamp_gate_metadata, ) -from gooddata_eval.core.agentic._outcome import agentic_detail, agentic_success, raise_agentic_failure +from gooddata_eval.core.agentic._outcome import ( + agentic_detail, + agentic_success, + chat_failure_exit, + gate_head, + label_runs, + log_eval_error_score, + raise_agentic_failure, +) from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -1021,7 +1029,7 @@ def _run_once(conv_id: str) -> AlertRunResult: partial_alert_id, _, partial_tool_called = _extract_alert_call(partial.tool_call_events or []) if partial_tool_called: alert_id_to_delete = partial_alert_id - exit_reason = LoopExit.CHAT_ERROR + exit_reason = chat_failure_exit(exc) break # `turns_used` counts attempts (set above, so a failed send still shows the # loop reached this iteration); `turns`/`steps` count completed work, so they @@ -1164,6 +1172,11 @@ def _run_detail(run: AlertRunResult) -> dict: } +def _failed_checks(run: AlertRunResult) -> list[dict]: + """The checks this run failed, by name. The arguments the run sent are in its detail.""" + return failed_check_records(FailedCheck(name) for name, ok in run.eval.checks().items() if not ok) + + def evaluate_agentic_alert_skill( host: str, token: str, @@ -1220,6 +1233,9 @@ def evaluate_agentic_alert_skill( user_context=user_context, ) + # Before the Langfuse scores, so an ERROR run gets its marker beside its usual scores. + run_labels = label_runs(summary.run_results, passed=lambda r: r.eval.strict_pass) + if langfuse is not None and dataset_item_id: # Pinned on the calling thread: a deferred poll must not widen its query window. window_end = utc_now() @@ -1231,6 +1247,8 @@ def _write_scores(ctx: RunTraceContext) -> None: pt = ctx.trace(run.conversation_id) strict_checks = run.eval.checks() with ctx.observe(pt, run_idx, conversation_id=run.conversation_id, output=strict_checks) as tid: + if (run_error := run_labels.errored_run(run_idx)) is not None: + log_eval_error_score(ctx, tid, run_error) for score_name, value in strict_checks.items(): ctx.score(tid, name=score_name, value=float(value), data_type="BOOLEAN") ctx.score(tid, name="turns", value=run.total_turns, data_type="NUMERIC") @@ -1288,10 +1306,11 @@ def _write_scores(ctx: RunTraceContext) -> None: summary.run_results, passed=lambda r: r.eval.strict_pass, detail=_run_detail, + failed_checks=_failed_checks, ) if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): - gate_note = gate_failure_note(gate, runs_passed, runs_effective) + gate_note = gate_head(gate, run_labels, runs_passed, failed_runs) raise_agentic_failure( AlertSkillAssertionError, f"Alert skill assertion failed. {gate_note} strict_pass={ev.strict_pass}. " @@ -1302,6 +1321,8 @@ def _write_scores(ctx: RunTraceContext) -> None: f"attributes_correct={ev.attributes_correct}, " f"granularity_correct={ev.granularity_correct}. " f"Actual args: {best.actual_alert_arguments}", + outcome=run_labels.outcome_record, + shadow=run_labels.shadow(gate), reasoning_steps=best.reasoning_steps, conversation_id=best.conversation_id, response_id=best.response_id, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/anomaly_detection.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/anomaly_detection.py index 25f2281a0..6897d68d1 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/anomaly_detection.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/anomaly_detection.py @@ -36,12 +36,19 @@ from gooddata_eval.core.agentic._gate import ( DEFAULT_GATE, EvalGate, - gate_failure_note, gate_passed, log_gate_scores, stamp_gate_metadata, ) -from gooddata_eval.core.agentic._outcome import agentic_detail, agentic_success, raise_agentic_failure +from gooddata_eval.core.agentic._outcome import ( + agentic_detail, + agentic_success, + chat_failure_exit, + gate_head, + label_runs, + log_eval_error_score, + raise_agentic_failure, +) from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -421,7 +428,7 @@ def _accumulate(result: ChatResult) -> None: _accumulate(partial) viz_args, execute_result = _extract_anomaly_calls(all_tool_call_events) turn_completed = False - exit_reason = LoopExit.CHAT_ERROR + exit_reason = chat_failure_exit(exc) break reasoning_steps.extend(chat_result.reasoning_steps or []) response_id = chat_result.response_id or response_id @@ -583,6 +590,9 @@ def evaluate_agentic_anomaly_detection( user_context=user_context, ) + # Before the Langfuse scores, so an ERROR run gets its marker beside its usual scores. + run_labels = label_runs(summary.run_results, passed=lambda r: r.evaluation.strict_pass) + if langfuse is not None and dataset_item_id: # Pinned on the calling thread: a deferred poll must not widen its query window. window_end = utc_now() @@ -613,6 +623,8 @@ def _write_scores(ctx: RunTraceContext) -> None: } ) with ctx.observe(pt, run_idx, conversation_id=run.conversation_id) as tid: + if (run_error := run_labels.errored_run(run_idx)) is not None: + log_eval_error_score(ctx, tid, run_error) for score_name, value in strict_checks.items(): ctx.score(tid, name=score_name, value=float(value), data_type="BOOLEAN") log_gate_scores(ctx, tid, gate=gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k) @@ -660,7 +672,7 @@ def _write_scores(ctx: RunTraceContext) -> None: runs_passed = sum(1 for r in summary.run_results if r.evaluation.strict_pass) if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): - gate_note = gate_failure_note(gate, runs_passed, len(summary.run_results)) + gate_note = gate_head(gate, run_labels, runs_passed, failed_runs) message = ( f"Anomaly detection assertion failed. {gate_note} strict_pass={ev.strict_pass} " f"(triggered={ev.triggered}, executed={ev.executed}, success={ev.success}, " @@ -672,6 +684,8 @@ def _write_scores(ctx: RunTraceContext) -> None: raise_agentic_failure( AnomalyDetectionAssertionError, message, + outcome=run_labels.outcome_record, + shadow=run_labels.shadow(gate), reasoning_steps=best.reasoning_steps, conversation_id=best.conversation_id, response_id=best.response_id, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py index 92f106340..5d227c9e8 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py @@ -56,7 +56,13 @@ summarize_visualizations, ) from gooddata_eval.core.agentic._gate import log_gate_scores -from gooddata_eval.core.agentic._outcome import agentic_detail, agentic_success, raise_agentic_failure +from gooddata_eval.core.agentic._outcome import ( + CHAT_FAILURE_EXITS, + agentic_detail, + agentic_success, + chat_failure_exit, + raise_agentic_failure, +) from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -1038,7 +1044,7 @@ def run_agentic_conversation( conversation_reasoning_step_events.extend(partial.reasoning_step_events or []) reasoning_steps.extend(partial.reasoning_steps or []) response_id = partial.response_id or response_id - turn_exit = LoopExit.CHAT_ERROR + turn_exit = chat_failure_exit(exc) break final_result = chat_result total_steps += chat_result.reasoning_step_count @@ -1255,7 +1261,7 @@ def run_agentic_conversation( # A chat fault used to escape the whole run, so reaching here did mean no # error. Now that it is caught and recorded, this has to read it back -- # otherwise a turn whose chat call failed reports no_error=True. - no_error=turn_exit is not LoopExit.CHAT_ERROR, + no_error=turn_exit not in CHAT_FAILURE_EXITS, activated_skills=declared or [], active_skills=sorted(active_skills), clarification_turns_used=clarification_turns, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/dashboard_skill.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/dashboard_skill.py index b7f2739f9..8449830e0 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/dashboard_skill.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/dashboard_skill.py @@ -13,16 +13,24 @@ from gooddata_eval.core.agentic._clarification import all_questions from gooddata_eval.core.agentic._conversation_context import CONFIRMATION_REPLY, asks_in_text, classify_reply, clip from gooddata_eval.core.agentic._failed_runs import build_failed_runs +from gooddata_eval.core.agentic._failure_message import FailedCheck, failed_check_records from gooddata_eval.core.agentic._filters import describe_selection, same_unordered_values, selection_of from gooddata_eval.core.agentic._gate import ( DEFAULT_GATE, EvalGate, - gate_failure_note, gate_passed, log_gate_scores, stamp_gate_metadata, ) -from gooddata_eval.core.agentic._outcome import agentic_detail, agentic_success, raise_agentic_failure +from gooddata_eval.core.agentic._outcome import ( + agentic_detail, + agentic_success, + chat_failure_exit, + gate_head, + label_runs, + log_eval_error_score, + raise_agentic_failure, +) from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -2302,7 +2310,7 @@ def _execute_single_dashboard_run( all_tool_call_events.extend(partial.tool_call_events or []) all_reasoning_step_events.extend(partial.reasoning_step_events or []) steps += partial.reasoning_step_count - exit_reason = LoopExit.CHAT_ERROR + exit_reason = chat_failure_exit(exc) break agent_elapsed = time.monotonic() - agent_started timings.agent_s += agent_elapsed @@ -2454,10 +2462,17 @@ def _run_detail(run: DashboardRunResult) -> dict[str, Any]: **run.evaluation.diagnostics, "failures": run.evaluation.failures, "notes": run.evaluation.notes, + # Why the loop stopped -- see LoopExit. + "exit_reason": run.exit_reason.value, "latency_breakdown": build_latency_breakdown(run.tool_call_events, run.reasoning_step_events), } +def _failed_checks(run: DashboardRunResult) -> list[dict]: + """The checks this run failed, by name. What each one found is in the run's ``failures``.""" + return failed_check_records(FailedCheck(name) for name, ok in run.evaluation.strict_checks.items() if not ok) + + def run_agentic_dashboard_skill( host: str, token: str, @@ -2580,6 +2595,9 @@ def evaluate_agentic_dashboard_skill( user_context=user_context, ) + # Before the Langfuse scores, so an ERROR run gets its marker beside its usual scores. + run_labels = label_runs(summary.run_results, passed=lambda r: r.evaluation.strict_pass) + if langfuse is not None and dataset_item_id: # Pinned on the calling thread: a deferred poll must not widen its query window. window_end = utc_now() @@ -2591,6 +2609,8 @@ def _write_scores(ctx: RunTraceContext) -> None: pt = ctx.trace(run.conversation_id) strict_checks = run.evaluation.strict_checks with ctx.observe(pt, run_idx, conversation_id=run.conversation_id, output=strict_checks) as tid: + if (run_error := run_labels.errored_run(run_idx)) is not None: + log_eval_error_score(ctx, tid, run_error) for score_name, value in strict_checks.items(): ctx.score(tid, name=score_name, value=float(value), data_type="BOOLEAN") for name, value in run.evaluation.diagnostics.items(): @@ -2642,10 +2662,11 @@ def _write_scores(ctx: RunTraceContext) -> None: summary.run_results, passed=lambda r: r.evaluation.strict_pass, detail=_run_detail, + failed_checks=_failed_checks, ) if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): - gate_note = gate_failure_note(gate, runs_passed, runs_effective) + gate_note = gate_head(gate, run_labels, runs_passed, failed_runs) # Two failures wear the same missing skill, and the advice differs. If the other # dashboard skill was activated, the flag that registers both is plainly on and the # model simply routed the wrong way. If neither was, the flag is off or the agent has @@ -2669,6 +2690,8 @@ def _write_scores(ctx: RunTraceContext) -> None: f"Checks: {best.evaluation.strict_checks}. " f"Failures: {'; '.join(best.evaluation.failures) or 'none reported'}." + (f" Notes (not scored): {notes}." if notes else ""), + outcome=run_labels.outcome_record, + shadow=run_labels.shadow(gate), reasoning_steps=best.reasoning_steps, conversation_id=best.conversation_id, response_id=best.response_id, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py index 1b69e2175..96f3c3e9a 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py @@ -20,11 +20,11 @@ from gooddata_eval.core.agentic._gate import ( DEFAULT_GATE, EvalGate, - gate_failure_note, gate_passed, log_gate_scores, stamp_gate_metadata, ) +from gooddata_eval.core.agentic._outcome import attach_labels, gate_head, label_runs from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -1856,6 +1856,9 @@ def _write_scores(ctx: RunTraceContext) -> None: unscored = summary.unscored_errors judge_errors = summary.judge_errors request_errors = summary.request_errors + # A run the judge or scorer could not grade is labelled ERROR (judge_failed); a run whose + # request failed is on the product path and stays FAIL. The gate is unchanged. + run_labels = label_runs(summary.run_results, passed=lambda r: r.evaluation.strict_pass) if not summary.scored_run_results: causes = [ *([f"judge returned no readable verdict for {len(judge_errors)} run(s)"] if judge_errors else []), @@ -1866,6 +1869,7 @@ def _write_scores(ctx: RunTraceContext) -> None: ) exc_judge.timings = item_timings exc_judge.run_latencies_s = [r.run_latency_s for r in summary.run_results] + attach_labels(exc_judge, run_labels, gate) raise exc_judge runs_passed = sum(1 for r in summary.run_results if r.evaluation.strict_pass) @@ -1899,7 +1903,7 @@ def _run_detail(run: DocumentRunResult) -> dict[str, Any]: failed_runs = build_failed_runs(summary.run_results, passed=lambda r: r.evaluation.strict_pass, detail=_run_detail) if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): - gate_note = gate_failure_note(gate, runs_passed, runs_effective, len(unscored)) + gate_note = gate_head(gate, run_labels, runs_passed, failed_runs, len(unscored)) skill_note = "" if not best.evaluation.wire_names_current: skill_note = " gen-ai still sends the Report names, so it predates the Document rename." @@ -1923,6 +1927,7 @@ def _run_detail(run: DocumentRunResult) -> dict[str, Any]: exc.runs_effective = runs_effective exc.failed_runs = failed_runs exc.run_latencies_s = [r.run_latency_s for r in summary.run_results] + attach_labels(exc, run_labels, gate) raise exc return AgenticEvalOutcome( runs_passed=runs_passed, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/general_question.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/general_question.py index 907f0567a..c1e3c8e87 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/general_question.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/general_question.py @@ -10,12 +10,18 @@ from gooddata_eval.core.agentic._gate import ( DEFAULT_GATE, EvalGate, - gate_failure_note, gate_passed, log_gate_scores, stamp_gate_metadata, ) -from gooddata_eval.core.agentic._outcome import agentic_detail, agentic_success, raise_agentic_failure +from gooddata_eval.core.agentic._outcome import ( + agentic_detail, + agentic_success, + attach_labels, + gate_head, + label_runs, + raise_agentic_failure, +) from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -417,6 +423,8 @@ def _write_scores(ctx: RunTraceContext) -> None: passed=lambda r: r.judge_error is None and r.passed, detail=_run_detail, ) + # A run the judge could not grade is labelled ERROR (judge_failed); the gate is unchanged. + run_labels = label_runs(summary.run_results, passed=lambda r: r.judge_error is None and r.passed) if not summary.scored_run_results: # Not one run produced a readable verdict, so this item has no result -- an error, @@ -431,13 +439,16 @@ def _write_scores(ctx: RunTraceContext) -> None: error, best, detail, failed_runs, runs_passed, runs_effective, run_latencies_s, run_latency_s(best.timings) ) error.timings = item_timings + attach_labels(error, run_labels, gate) raise error if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): - gate_note = gate_failure_note(gate, runs_passed, runs_effective, len(unscored)) + gate_note = gate_head(gate, run_labels, runs_passed, failed_runs, len(unscored)) raise_agentic_failure( GeneralQuestionAssertionError, f"General question assertion failed. {gate_note} passed={best.passed}. Reasoning: {best.reasoning}", + outcome=run_labels.outcome_record, + shadow=run_labels.shadow(gate), reasoning_steps=best.reasoning_steps, conversation_id=best.conversation_id, response_id=best.response_id, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/guardrail.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/guardrail.py index f9e281c2d..64d835ab8 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/guardrail.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/guardrail.py @@ -10,12 +10,18 @@ from gooddata_eval.core.agentic._gate import ( DEFAULT_GATE, EvalGate, - gate_failure_note, gate_passed, log_gate_scores, stamp_gate_metadata, ) -from gooddata_eval.core.agentic._outcome import agentic_detail, agentic_success, raise_agentic_failure +from gooddata_eval.core.agentic._outcome import ( + agentic_detail, + agentic_success, + attach_labels, + gate_head, + label_runs, + raise_agentic_failure, +) from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -375,6 +381,8 @@ def _write_scores(ctx: RunTraceContext) -> None: passed=lambda r: r.judge_error is None and r.passed, detail=_run_detail, ) + # A run the judge could not grade is labelled ERROR (judge_failed); the gate is unchanged. + run_labels = label_runs(summary.run_results, passed=lambda r: r.judge_error is None and r.passed) if not summary.scored_run_results: # No readable verdict for any run: an error, not K failures. Raised after the @@ -386,13 +394,16 @@ def _write_scores(ctx: RunTraceContext) -> None: _attach_diagnostics( error, best, detail, failed_runs, runs_passed, runs_effective, run_latencies_s, best.run_latency_s ) + attach_labels(error, run_labels, gate) raise error if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): - gate_note = gate_failure_note(gate, runs_passed, runs_effective, len(unscored)) + gate_note = gate_head(gate, run_labels, runs_passed, failed_runs, len(unscored)) raise_agentic_failure( GuardrailAssertionError, f"Guardrail assertion failed. {gate_note} passed={best.passed}. Reasoning: {best.reasoning}", + outcome=run_labels.outcome_record, + shadow=run_labels.shadow(gate), reasoning_steps=best.reasoning_steps, conversation_id=best.conversation_id, response_id=best.response_id, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py index 683f8d42d..6aea15f50 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py @@ -14,12 +14,19 @@ from gooddata_eval.core.agentic._gate import ( DEFAULT_GATE, EvalGate, - gate_failure_note, gate_passed, log_gate_scores, stamp_gate_metadata, ) -from gooddata_eval.core.agentic._outcome import agentic_detail, agentic_success, raise_agentic_failure +from gooddata_eval.core.agentic._outcome import ( + agentic_detail, + agentic_success, + chat_failure_exit, + gate_head, + label_runs, + log_eval_error_score, + raise_agentic_failure, +) from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -329,7 +336,7 @@ def _accumulate(result: ChatResult) -> None: if create_args is not None: turn_wall_clock_sec = partial.turn_wall_clock_sec turn_completed = False - exit_reason = LoopExit.CHAT_ERROR + exit_reason = chat_failure_exit(exc) break turns += 1 steps += chat_result.reasoning_step_count @@ -499,6 +506,9 @@ def evaluate_agentic_kda_skill( user_context=user_context, ) + # Before the Langfuse scores, so an ERROR run gets its marker beside its usual scores. + run_labels = label_runs(summary.run_results, passed=lambda r: r.evaluation.strict_pass) + if langfuse is not None and dataset_item_id: # Pinned on the calling thread: a deferred poll must not widen its query window. window_end = utc_now() @@ -525,6 +535,8 @@ def _write_scores(ctx: RunTraceContext) -> None: "[kda-report] %s: strict_pass=%s latency_sec=%s", run_name, ev.strict_pass, turn_wall_clock_sec ) with ctx.observe(pt, run_idx, conversation_id=run.conversation_id, output=strict_checks) as tid: + if (run_error := run_labels.errored_run(run_idx)) is not None: + log_eval_error_score(ctx, tid, run_error) for score_name, value in strict_checks.items(): ctx.score(tid, name=score_name, value=float(value), data_type="BOOLEAN") ctx.score(tid, name="kda_disambiguated", value=float(ev.disambiguated), data_type="BOOLEAN") @@ -590,7 +602,7 @@ def _write_scores(ctx: RunTraceContext) -> None: ) if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): - gate_note = gate_failure_note(gate, runs_passed, runs_effective) + gate_note = gate_head(gate, run_labels, runs_passed, failed_runs) message = ( f"KDA skill assertion failed. {gate_note} strict_pass={ev.strict_pass} " f"(triggered={ev.triggered}, executed={ev.executed}, " @@ -601,6 +613,8 @@ def _write_scores(ctx: RunTraceContext) -> None: raise_agentic_failure( KdaSkillAssertionError, message, + outcome=run_labels.outcome_record, + shadow=run_labels.shadow(gate), reasoning_steps=best.reasoning_steps, conversation_id=best.conversation_id, response_id=best.response_id, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/metric_skill.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/metric_skill.py index f662b205b..a00586702 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/metric_skill.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/metric_skill.py @@ -13,15 +13,23 @@ from gooddata_sdk import GoodDataSdk from gooddata_eval.core.agentic._failed_runs import build_failed_runs +from gooddata_eval.core.agentic._failure_message import FailedCheck, failed_check_records from gooddata_eval.core.agentic._gate import ( DEFAULT_GATE, EvalGate, - gate_failure_note, gate_passed, log_gate_scores, stamp_gate_metadata, ) -from gooddata_eval.core.agentic._outcome import agentic_detail, agentic_success, raise_agentic_failure +from gooddata_eval.core.agentic._outcome import ( + agentic_detail, + agentic_success, + chat_failure_exit, + gate_head, + label_runs, + log_eval_error_score, + raise_agentic_failure, +) from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -352,7 +360,7 @@ def _execute_single_metric_run( for metric_id in _extract_created_metric_ids(partial.tool_call_events or []): if metric_id not in created_metric_ids: created_metric_ids.append(metric_id) - exit_reason = LoopExit.CHAT_ERROR + exit_reason = chat_failure_exit(exc) break agent_elapsed = time.monotonic() - agent_started timings.agent_s += agent_elapsed @@ -562,6 +570,9 @@ def evaluate_agentic_metric_skill( user_context=user_context, ) + # Before the Langfuse scores, so an ERROR run gets its marker beside its usual scores. + run_labels = label_runs(summary.run_results, passed=lambda r: r.metric_created and r.maql_correct) + if langfuse is not None and dataset_item_id: # Pinned on the calling thread: a deferred poll must not widen its query window. window_end = utc_now() @@ -577,6 +588,8 @@ def _write_scores(ctx: RunTraceContext) -> None: conversation_id=run.conversation_id, output={"metric_created": run.metric_created, "maql_correct": run.maql_correct}, ) as tid: + if (run_error := run_labels.errored_run(run_idx)) is not None: + log_eval_error_score(ctx, tid, run_error) ctx.score(tid, name="metric_created", value=float(run.metric_created), data_type="BOOLEAN") ctx.score(tid, name="maql_correct", value=float(run.maql_correct), data_type="BOOLEAN") ctx.score(tid, name="turns", value=run.total_turns, data_type="NUMERIC") @@ -631,6 +644,21 @@ def _run_detail(run: MetricRunResult) -> dict: **timeline_detail(run.tool_call_events, run.reasoning_step_events), } + def _failed_checks(run: MetricRunResult) -> list[dict]: + """The checks this run failed; the MAQL pair is the whole diagnosis for this kind.""" + checks: list[FailedCheck] = [] + if not run.metric_created: + checks.append(FailedCheck("metric_created", expected=True, actual=False)) + if not run.maql_correct: + checks.append( + FailedCheck( + "maql_correct", + expected=[c.get("maql", "") for c in expected_outputs_list], + actual=run.actual_maql, + ) + ) + return failed_check_records(checks) + detail = agentic_detail( best.tool_call_events, best.reasoning_step_events, @@ -650,10 +678,11 @@ def _run_detail(run: MetricRunResult) -> dict: summary.run_results, passed=lambda r: r.metric_created and r.maql_correct, detail=_run_detail, + failed_checks=_failed_checks, ) if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): - gate_note = gate_failure_note(gate, runs_passed, runs_effective) + gate_note = gate_head(gate, run_labels, runs_passed, failed_runs) candidates_str = "; ".join(repr(c.get("maql", "")) for c in expected_outputs_list) raise_agentic_failure( MetricSkillAssertionError, @@ -661,6 +690,8 @@ def _run_detail(run: MetricRunResult) -> dict: f"metric_created={best.metric_created}, maql_correct={best.maql_correct}. " f"Expected MAQL (candidates): {candidates_str}. " f"Actual MAQL: {best.actual_maql}.", + outcome=run_labels.outcome_record, + shadow=run_labels.shadow(gate), reasoning_steps=best.reasoning_steps, conversation_id=best.conversation_id, response_id=best.response_id, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py index b06821452..a39b3eec2 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py @@ -12,15 +12,24 @@ from dataclasses import dataclass, field, replace from gooddata_eval.core.agentic._failed_runs import build_failed_runs +from gooddata_eval.core.agentic._failure_message import FailedCheck, failed_check_records from gooddata_eval.core.agentic._gate import ( DEFAULT_GATE, EvalGate, - gate_failure_note, gate_passed, log_gate_scores, stamp_gate_metadata, ) -from gooddata_eval.core.agentic._outcome import agentic_detail, agentic_success, raise_agentic_failure +from gooddata_eval.core.agentic._outcome import ( + CHAT_FAILURE_EXITS, + agentic_detail, + agentic_success, + chat_failure_exit, + gate_head, + label_runs, + log_eval_error_score, + raise_agentic_failure, +) from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -204,6 +213,7 @@ def _execute_single_run( max_iterations: int = _DEFAULT_MAX_ITERATIONS, requires_execution: bool = False, options: ExpectationOptions | None = None, + user_context: dict | None = None, ) -> RunResult: """Drive one full multi-turn conversation and evaluate the result.""" options = _effective_options(options, requires_execution) @@ -232,21 +242,21 @@ def _execute_single_run( current_result = client.send_message(conversation_id, question) except ChatError as exc: # The opening request, so there is no partial conversation to evaluate: fall through - # to the empty-result path with CHAT_ERROR recorded. Raising here would discard every + # to the empty-result path with its exit recorded. Raising here would discard every # K-run already completed along with any exit_reason. print(f"[CHAT] send_message failed for conversation {conversation_id}: {exc}") current_result = getattr(exc, "partial_result", None) or ChatResult() - # The loop that collects events does not run after CHAT_ERROR, so a partial result's + # The loop that collects events does not run after a failed opening request, so a partial result's # chart executions are added here. all_tool_call_events.extend(current_result.tool_call_events) - exit_reason = LoopExit.CHAT_ERROR + exit_reason = chat_failure_exit(exc) # Counted here, not at the top of the loop: this request is sent unconditionally, so # with max_iterations=0 the loop body never runs and total_turns would report 0 turns # for a conversation the agent did receive. The loop's own increment is skipped on its # first pass to compensate, keeping total_turns == number of send_message calls. total_turns += 1 - for iteration in range(max_iterations if exit_reason is not LoopExit.CHAT_ERROR else 0): + for iteration in range(max_iterations if exit_reason not in CHAT_FAILURE_EXITS else 0): if iteration: total_turns += 1 total_steps += current_result.reasoning_step_count @@ -297,7 +307,7 @@ def _execute_single_run( # The loop adds a turn's events at its next pass, which a break skips. The # partial result is scored, so its chart executions count too. all_tool_call_events.extend(partial.tool_call_events) - exit_reason = LoopExit.CHAT_ERROR + exit_reason = chat_failure_exit(exc) break skill_activated = _check_visualization_skill_activated(all_tool_call_events) @@ -305,7 +315,9 @@ def _execute_single_run( if current_result.created_visualizations and current_result.created_visualizations.objects: actual_output = current_result.created_visualizations.objects[0] - eval_result, best_expected = _evaluate_against_candidates(expected_outputs, actual_output, skill_activated) + eval_result, best_expected = _evaluate_against_candidates( + expected_outputs, actual_output, skill_activated, user_context=user_context + ) eval_result = with_execution( eval_result, all_tool_call_events, current_result.text_response, options.requires_execution ) @@ -316,7 +328,7 @@ def _execute_single_run( actual_output, all_tool_call_events, answer_text, - interrupted=exit_reason is LoopExit.CHAT_ERROR, + interrupted=exit_reason in CHAT_FAILURE_EXITS, ) return RunResult( @@ -392,7 +404,14 @@ def run_agentic_visualization( try: run_results.append( _execute_single_run( - client, conv_id_0, question, expected_outputs, max_iterations, requires_execution, options + client, + conv_id_0, + question, + expected_outputs, + max_iterations, + requires_execution, + options, + user_context=user_context, ) ) finally: @@ -404,7 +423,14 @@ def run_agentic_visualization( try: run_results.append( _execute_single_run( - client, conv_id, question, expected_outputs, max_iterations, requires_execution, options + client, + conv_id, + question, + expected_outputs, + max_iterations, + requires_execution, + options, + user_context=user_context, ) ) finally: @@ -445,6 +471,76 @@ def _run_detail(run: RunResult) -> dict: } +def _failed_checks(run: RunResult) -> list[dict]: + """The checks this run failed, by their ``detail`` names, with expected/actual where known. + + Follows ``EvaluationResult.strict_pass`` term by term, so the list is empty exactly when + the run passed. + """ + ev = run.eval_result + checks: list[FailedCheck] = [] + if ev.scored_without_chart: + if ev.visualization_created: + actual_type = run.actual_output.type if run.actual_output else None + checks.append(FailedCheck("no_chart", expected="no chart", actual=actual_type)) + if ev.answer_correct is False: + checks.append(FailedCheck("answer_correct", actual=ev.answer_problems)) + return failed_check_records(checks) + if not ev.visualization_created: + checks.append(FailedCheck("visualization_created", expected=True, actual=False)) + if not ev.cross_ref_valid: + checks.append(FailedCheck("cross_ref_valid", actual=ev.cross_ref_errors)) + if not ev.metrics_correct: + if ev.expected_metric_uris == ev.actual_metric_uris and ev.metric_layout_problems: + # The metric set matched; the order or axis did not, and only these say which. + checks.append(FailedCheck("metrics_correct", label="layout", actual=ev.metric_layout_problems)) + else: + checks.append( + FailedCheck( + "metrics_correct", expected=sorted(ev.expected_metric_uris), actual=sorted(ev.actual_metric_uris) + ) + ) + if not ev.dimensions_correct: + checks.append( + FailedCheck("dimensions_correct", expected=sorted(ev.expected_dim_uris), actual=sorted(ev.actual_dim_uris)) + ) + if not ev.filters_correct: + differing = [ + FailedCheck( + "filters_correct", + label=category, + expected=ev.expected_filters.get(category, []), + actual=ev.actual_filters.get(category, []), + ) + for category in sorted(set(ev.expected_filters) | set(ev.actual_filters)) + if ev.expected_filters.get(category, []) != ev.actual_filters.get(category, []) + # An inherited date filter was accepted, so it is not why the filters failed. + and not (category == "date" and ev.filter_date_inherited) + ] + checks.extend(differing or [FailedCheck("filters_correct")]) + if not ev.sorts_correct: + checks.append(FailedCheck("sorts_correct", expected=ev.expected_sorts, actual=ev.actual_sorts)) + if not ev.viz_type_hard: + checks.append( + FailedCheck( + "viz_type_hard", + expected=run.best_expected.type if run.best_expected else None, + actual=run.actual_output.type if run.actual_output else None, + ) + ) + if ev.requires_execution and not ev.executed: + checks.append(FailedCheck("executed", expected=True, actual=False)) + if ev.buckets_correct is False: + checks.append(FailedCheck("buckets_correct", expected=ev.expected_buckets, actual=ev.actual_buckets)) + if ev.answer_correct is False: + checks.append(FailedCheck("answer_correct", actual=ev.answer_problems)) + if ev.format_correct is False: + checks.append(FailedCheck("format_correct", actual=ev.format_problems)) + if ev.part_present is False: + checks.append(FailedCheck("part_present", expected=True, actual=False)) + return failed_check_records(checks) + + def _strict_checks(ev: EvaluationResult) -> dict[str, bool]: """The scores one run is gated on, by their Langfuse names. @@ -579,6 +675,9 @@ def evaluate_agentic_visualization( options=options, ) + # Before the Langfuse scores, so an ERROR run gets its marker beside its usual scores. + run_labels = label_runs(summary.run_results, passed=lambda r: r.eval_result.strict_pass) + if langfuse is not None and dataset_item_id: # Pinned on the calling thread: a deferred poll must not widen its query window. window_end = utc_now() @@ -592,6 +691,8 @@ def _write_scores(ctx: RunTraceContext) -> None: ev = run.eval_result strict_checks = _strict_checks(ev) with ctx.observe(pt, run_idx, conversation_id=run.conversation_id, output=strict_checks) as tid: + if (run_error := run_labels.errored_run(run_idx)) is not None: + log_eval_error_score(ctx, tid, run_error) for score_name, value in strict_checks.items(): # Scored below on every item, gating or not. if score_name != "assertion-vis-executed": @@ -664,10 +765,11 @@ def _write_scores(ctx: RunTraceContext) -> None: summary.run_results, passed=lambda r: r.eval_result.strict_pass, detail=_run_detail, + failed_checks=_failed_checks, ) if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): - gate_note = gate_failure_note(gate, runs_passed, runs_effective) + gate_note = gate_head(gate, run_labels, runs_passed, failed_runs) n = len(expected_outputs) candidate_note = f" (closest of {n} candidates)" if n > 1 else "" cross_ref_detail = (" → " + "; ".join(ev.cross_ref_errors)) if ev.cross_ref_errors else "" @@ -699,8 +801,9 @@ def _write_scores(ctx: RunTraceContext) -> None: f" expected : {sorted(ev.expected_dim_uris)}\n" f" actual : {sorted(ev.actual_dim_uris)}\n" f" Filters Correct : {ev.filters_correct}\n" - f" date : {ev.filter_date_score}\n" - f"{_filter_diff('date', ev)}" + f" date : {ev.filter_date_score}" + f"{' (inherited from dashboard context)' if ev.filter_date_inherited else ''}\n" + f"{'' if ev.filter_date_inherited else _filter_diff('date', ev)}" f" ranking : {ev.filter_ranking_score}\n" f"{_filter_diff('ranking', ev)}" f" attribute : {ev.filter_attribute_score}\n" @@ -711,6 +814,8 @@ def _write_scores(ctx: RunTraceContext) -> None: f"{_options_lines(ev, best.answer_text)}" f" Stated Value Matches : {ev.stated_value_matches} (reported only)\n" "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n", + outcome=run_labels.outcome_record, + shadow=run_labels.shadow(gate), reasoning_steps=best.reasoning_steps, conversation_id=best.conversation_id, response_id=best.response_id, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/what_if.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/what_if.py index de124ceed..a0ad72f8d 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/what_if.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/what_if.py @@ -30,12 +30,19 @@ from gooddata_eval.core.agentic._gate import ( DEFAULT_GATE, EvalGate, - gate_failure_note, gate_passed, log_gate_scores, stamp_gate_metadata, ) -from gooddata_eval.core.agentic._outcome import agentic_detail, agentic_success, raise_agentic_failure +from gooddata_eval.core.agentic._outcome import ( + agentic_detail, + agentic_success, + chat_failure_exit, + gate_head, + label_runs, + log_eval_error_score, + raise_agentic_failure, +) from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -378,7 +385,7 @@ def _accumulate(result: ChatResult) -> None: _accumulate(partial) create_args, execute_result = _extract_what_if_calls(all_tool_call_events) turn_completed = False - exit_reason = LoopExit.CHAT_ERROR + exit_reason = chat_failure_exit(exc) break reasoning_steps.extend(chat_result.reasoning_steps or []) response_id = chat_result.response_id or response_id @@ -544,6 +551,9 @@ def evaluate_agentic_what_if( user_context=user_context, ) + # Before the Langfuse scores, so an ERROR run gets its marker beside its usual scores. + run_labels = label_runs(summary.run_results, passed=lambda r: r.evaluation.strict_pass) + if langfuse is not None and dataset_item_id: # Pinned on the calling thread: a deferred poll must not widen its query window. window_end = utc_now() @@ -576,6 +586,8 @@ def _write_scores(ctx: RunTraceContext) -> None: } ) with ctx.observe(pt, run_idx, conversation_id=run.conversation_id) as tid: + if (run_error := run_labels.errored_run(run_idx)) is not None: + log_eval_error_score(ctx, tid, run_error) for score_name, value in strict_checks.items(): ctx.score(tid, name=score_name, value=float(value), data_type="BOOLEAN") log_gate_scores(ctx, tid, gate=gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k) @@ -624,7 +636,7 @@ def _write_scores(ctx: RunTraceContext) -> None: runs_passed = sum(1 for r in summary.run_results if r.evaluation.strict_pass) if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): - gate_note = gate_failure_note(gate, runs_passed, len(summary.run_results)) + gate_note = gate_head(gate, run_labels, runs_passed, failed_runs) message = ( f"What-if assertion failed. {gate_note} strict_pass={ev.strict_pass} " f"(triggered={ev.triggered}, executed={ev.executed}, success={ev.success}, " @@ -637,6 +649,8 @@ def _write_scores(ctx: RunTraceContext) -> None: raise_agentic_failure( WhatIfAssertionError, message, + outcome=run_labels.outcome_record, + shadow=run_labels.shadow(gate), reasoning_steps=best.reasoning_steps, conversation_id=best.conversation_id, response_id=best.response_id, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/config.py b/packages/gooddata-eval/src/gooddata_eval/core/config.py index bb5f838ae..6b4fce312 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/config.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/config.py @@ -17,6 +17,17 @@ def env_flag(name: str) -> bool: return os.environ.get(name, "").strip().lower() not in _ENV_FALSE +# Opt-in: accept a date filter the agent carried over from the dashboard open in +# the item's `user_context`, when the expected output states no date filter. Off by default: +# a caller that does not ask for it gets the strict date comparison. +ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR = "GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER" + + +def accept_context_date_filter() -> bool: + """Whether `check_filters` may accept a date filter inherited from the dashboard context.""" + return env_flag(ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR) + + # The judge model, overridable for comparison experiments. The default is deliberate on # two counts: gpt-4o honours temperature=0, so a given response always gets the same # verdict; and it is NOT the family under test, so the judge is not grading its own diff --git a/packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py b/packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py index 10806d047..59880c6dd 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py @@ -83,6 +83,15 @@ class EvaluationResult: format_correct: bool | None = None format_problems: list[str] = field(default_factory=list) part_present: bool | None = None + # Set only when `filter_date_score` holds because the actual date filter is the one the + # dashboard in the item's `user_context` applies (opt-in, see `check_filters`): the date + # dataset that filter used. Reported in the detail only when set. + filter_date_inherited_dataset: str | None = None + + @property + def filter_date_inherited(self) -> bool: + """Whether `filter_date_score` holds only through the dashboard-context rule.""" + return self.filter_date_inherited_dataset is not None @property def scored_without_chart(self) -> bool: @@ -453,6 +462,9 @@ def _evaluate_visualization( actual: CreatedVisualization | None, skill_activated: bool = False, today: date | None = None, + *, + user_context: Mapping | None = None, + accept_context_date_filter: bool | None = None, ) -> EvaluationResult: # One anchor for the scoring and for the filters reported beside it. check_filters # already shares an anchor across its two sides, but resolving the reported filters @@ -489,7 +501,13 @@ def _evaluate_visualization( cross_ref_valid, cross_ref_errors = validate_cross_references(actual) act_metric_uris = get_metric_uri_set(actual) act_dim_uris = get_dimension_uri_set(actual) - filter_scores = check_filters(expected, actual, today) + filter_scores = check_filters( + expected, + actual, + today, + user_context=user_context, + accept_context_date_filter=accept_context_date_filter, + ) metric_layout_problems = check_metric_layout(expected, actual) return EvaluationResult( visualization_created=True, @@ -516,6 +534,7 @@ def _evaluate_visualization( expected_buckets=bucket_layout(expected) if expected.buckets_strict else None, actual_buckets=bucket_layout(actual) if expected.buckets_strict else None, metric_layout_problems=metric_layout_problems, + filter_date_inherited_dataset=filter_scores.date_inherited_dataset, ) @@ -551,13 +570,29 @@ def _evaluate_against_candidates( actual: CreatedVisualization | None, skill_activated: bool = False, today: date | None = None, + *, + user_context: Mapping | None = None, + accept_context_date_filter: bool | None = None, ) -> tuple[EvaluationResult, CreatedVisualization | None]: if not expected_outputs: return _no_candidate_result(actual, skill_activated), None # Anchored once here too: the candidates are ranked against each other, so letting # each resolve its own today could rank them on spans from different days. today = today or date.today() - pairs = [(_evaluate_visualization(exp, actual, skill_activated, today), exp) for exp in expected_outputs] + pairs = [ + ( + _evaluate_visualization( + exp, + actual, + skill_activated, + today, + user_context=user_context, + accept_context_date_filter=accept_context_date_filter, + ), + exp, + ) + for exp in expected_outputs + ] best_result, best_expected = max(pairs, key=lambda p: (p[0].strict_pass, p[0].strict_checks_passed_count)) return best_result, best_expected @@ -624,6 +659,13 @@ def evaluation_result_detail(ev: EvaluationResult) -> dict: "actual_dim_uris": sorted(ev.actual_dim_uris), "expected_filters": ev.expected_filters, "actual_filters": ev.actual_filters, + # Nested and only when it applies: `quality_score` counts every top-level boolean, and + # a run scored without the opt-in rule must not carry the key at all. + **( + {"filter_date_context": {"inherited": True, "dataset": ev.filter_date_inherited_dataset}} + if ev.filter_date_inherited + else {} + ), "expected_sorts": ev.expected_sorts, "actual_sorts": ev.actual_sorts, **({"metric_layout_problems": ev.metric_layout_problems} if ev.metric_layout_problems else {}), @@ -658,7 +700,9 @@ def evaluate(self, item: DatasetItem, chat_result: ChatResult) -> ItemEvaluation candidates = _parse_expected(item.expected_output) actual = _extract_actual(chat_result) skill_activated = _check_visualization_skill_activated(chat_result.tool_call_events) - ev, _best_expected = _evaluate_against_candidates(candidates, actual, skill_activated) + ev, _best_expected = _evaluate_against_candidates( + candidates, actual, skill_activated, user_context=item.user_context + ) ev = with_execution(ev, chat_result.tool_call_events, chat_result.text_response, options.requires_execution) ev = with_options(ev, options, actual, chat_result.tool_call_events, render_answer_text(chat_result)) return ItemEvaluation( diff --git a/packages/gooddata-eval/src/gooddata_eval/core/models.py b/packages/gooddata-eval/src/gooddata_eval/core/models.py index 094cee08e..d3ef0364b 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/models.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/models.py @@ -28,6 +28,11 @@ class LoopExit(str, Enum): This does not change any verdict -- an exhausted run still fails. It makes the three cases countable, so "is the turn budget too tight" becomes a question the data can answer rather than one that has to be argued. + + Which members label a run ERROR (the harness failed) rather than FAIL is decided in one + place, ``core/outcome.py``. The label decides no verdict. Only SIMULATED_USER_FAILED is an + ERROR: CHAT_ERROR, AGENT_TIMEOUT and TURN_INCOMPLETE are on the product path, and a + customer hits them too. """ SUCCESS = "success" @@ -48,16 +53,27 @@ class LoopExit(str, Enum): SIMULATED_USER_FAILED = "simulated_user_failed" """The harness's own simulated-user model failed to produce a follow-up. A harness-side - fault, not an agent one -- reporting should treat it as an error, not a scored failure.""" + fault, not an agent one, so the run is labelled ERROR (``core/outcome.py``).""" CHAT_ERROR = "chat_error" - """The chat call itself raised mid-conversation. Like SIMULATED_USER_FAILED this is - infrastructure rather than agent capability, but it comes from the GoodData side.""" + """The chat call itself raised mid-conversation and was not one of the agent's own + failures below: a gen-ai error event, a broken stream, or a 5xx still failing after its + retries. It is on the product path -- a customer would see it too -- so the run is a + FAIL, not a harness ERROR.""" + + AGENT_TIMEOUT = "agent_timeout" + """The agent exceeded the per-turn or per-item wall-clock budget (``TurnTimeoutError``). + The agent's own failure to answer in time, so the run is a FAIL.""" + + TURN_INCOMPLETE = "turn_incomplete" + """The agent ended its turn without a final answer, e.g. at its own iteration limit + (``TurnIncompleteError``). The agent's own failure, so the run is a FAIL.""" NOT_RUN = "not_run" """The loop never started -- e.g. an agentic_conversation turn whose $ref to an earlier turn's output could not be resolved, so the turn was skipped before any message was - sent. Distinct from every other member, which all imply at least one turn happened.""" + sent. Distinct from every other member, which all imply at least one turn happened. A + FAIL: its cause is usually the earlier turn's failure, which the item already reports.""" class AacQueryField(BaseModel): @@ -441,6 +457,12 @@ class AgenticAssertionError(AssertionError): failed_runs: list[dict] run_latencies_s: list[float] best_run_latency_s: float | None + outcome: dict + """``{"status": "error", "reason": ...}``, set only when every run of the item errored + (``core/outcome.py``): the item is a harness error, not an agent failure. A runner labels + it; it stays a failure.""" + shadow: dict + """The phase-2 candidate verdict (``core.outcome.shadow_verdict``). Recorded, never used.""" class AgenticEvalOutcome(BaseModel): diff --git a/packages/gooddata-eval/src/gooddata_eval/core/outcome.py b/packages/gooddata-eval/src/gooddata_eval/core/outcome.py new file mode 100644 index 000000000..539ca5267 --- /dev/null +++ b/packages/gooddata-eval/src/gooddata_eval/core/outcome.py @@ -0,0 +1,143 @@ +# (C) 2026 GoodData Corporation +"""One label per run: PASS, FAIL, or ERROR when the test harness, not the agent, failed. + +A run can end in a way that is the harness's fault: the simulated user's own model call +fails, or the judge cannot grade the answer. Scored as usual, such a run reads as an agent +failure. ``classify_run`` names the difference, in one place for every kind. + +The labels decide no verdict. A run labelled ERROR still counts as a run that did not pass, +so every gate, pass rate and trend is what it was before labels existed. The labels only +tell a reader why a run did not pass (QA-30213, phase 1). ``shadow_verdict`` records what a +candidate rule that leaves ERROR runs out of the gate would have given, so that rule can be +judged on real nights before anyone adopts it (QA-30296). + +The rule is fail-closed. A run is ERROR only for a cause on ``_ERROR_EXITS`` or a judge +failure: the agent never had a fair chance, and a customer would not have hit the cause. +Anything else, including a chat API error and an unknown exit, is FAIL. +""" + +from __future__ import annotations + +import math +from collections import Counter +from collections.abc import Sequence +from dataclasses import dataclass +from enum import Enum +from typing import Any + +from gooddata_eval.core.config import normalize_gate +from gooddata_eval.core.models import LoopExit + +__all__ = [ + "EVAL_ERROR_SCORE", + "JUDGE_FAILED", + "Outcome", + "OutcomeStatus", + "classify_run", + "item_outcome", + "shadow_verdict", +] + + +class OutcomeStatus(str, Enum): + """What one run, or one item, amounts to.""" + + PASS = "pass" + FAIL = "fail" + ERROR = "error" + """The test harness failed, so the run says nothing about the agent.""" + + +@dataclass(frozen=True) +class Outcome: + """A status and a machine-readable reason, e.g. ``simulated_user_failed``.""" + + status: OutcomeStatus + reason: str | None = None + + @classmethod + def passed(cls) -> Outcome: + return cls(OutcomeStatus.PASS) + + @classmethod + def failed(cls, reason: str | None = None) -> Outcome: + return cls(OutcomeStatus.FAIL, reason) + + @classmethod + def error(cls, reason: str) -> Outcome: + return cls(OutcomeStatus.ERROR, reason) + + @property + def is_error(self) -> bool: + return self.status is OutcomeStatus.ERROR + + def as_record(self) -> dict[str, Any]: + """The JSON-ready form written into ``failed_runs[].outcome`` and onto the exception.""" + return {"status": self.status.value, "reason": self.reason} + + +EVAL_ERROR_SCORE = "eval_error" +"""The extra Langfuse score (NUMERIC, 1) an ERROR run gets, its comment the reason. The +run's other scores are written as always.""" + +JUDGE_FAILED = "judge_failed" +"""The reason of a run the judge or scorer could not grade.""" + +# The only loop exits that make a run an ERROR. A chat API error stays a FAIL: it is on the +# product path, and a customer would hit it too. +_ERROR_EXITS: frozenset[LoopExit] = frozenset({LoopExit.SIMULATED_USER_FAILED}) + + +def classify_run(exit_reason: LoopExit | str | None, *, passed: bool, judge_error: str | None = None) -> Outcome: + """The label of one run, from whether it passed, why its loop stopped and its judge. + + A run that passed is PASS whatever its exit: its checks held on what the agent produced. + A run whose judge failed is ERROR (``judge_failed``), and so is a run whose exit is on + ``_ERROR_EXITS``. Every other run is FAIL, its reason the exit (or None for a kind whose + loop records none). An exit string that is not a ``LoopExit`` value is kept as the reason + of a FAIL rather than raising: an unknown cause is never an error. + """ + if passed: + return Outcome.passed() + if judge_error is not None: + return Outcome.error(JUDGE_FAILED) + loop_exit: LoopExit | None + if isinstance(exit_reason, LoopExit) or exit_reason is None: + loop_exit = exit_reason + else: + try: + loop_exit = LoopExit(exit_reason) + except ValueError: + return Outcome.failed(exit_reason) + if loop_exit in _ERROR_EXITS: + return Outcome.error(loop_exit.value) + return Outcome.failed(loop_exit.value if loop_exit is not None else None) + + +def item_outcome(outcomes: Sequence[Outcome]) -> Outcome | None: + """ERROR when every run of the item is ERROR, with the most frequent reason; else None. + + None means the item keeps the verdict its gate gave it. An item with some ERROR runs is + not an ERROR: its other runs were scored, and its message names the ERROR runs. + """ + if not outcomes or not all(o.is_error for o in outcomes): + return None + reasons = Counter(o.reason or "unknown" for o in outcomes) + return Outcome.error(max(sorted(reasons), key=lambda reason: reasons[reason])) + + +def shadow_verdict(outcomes: Sequence[Outcome], gate: str | None) -> dict[str, Any]: + """What the candidate phase-2 rule would give this item. Recorded, never used. + + The rule leaves ERROR runs out of K. With fewer than ceil(K/2) scored runs left the item + is ``error``; otherwise ``gate`` is applied to the scored runs. + """ + scored = [o for o in outcomes if not o.is_error] + passes = sum(1 for o in scored if o.status is OutcomeStatus.PASS) + if not scored or len(scored) < math.ceil(len(outcomes) / 2): + verdict = "error" + elif normalize_gate(gate) == "power": + verdict = "pass" if passes == len(scored) else "fail" + else: + verdict = "pass" if passes else "fail" + return {"verdict": verdict, "runs_scored": len(scored), "runs_total": len(outcomes)} diff --git a/packages/gooddata-eval/src/gooddata_eval/core/scoring.py b/packages/gooddata-eval/src/gooddata_eval/core/scoring.py index 07e6c389e..b6fbc99cd 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/scoring.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/scoring.py @@ -7,6 +7,7 @@ from dataclasses import dataclass from datetime import date, timedelta +from gooddata_eval.core.config import accept_context_date_filter as _accept_context_date_filter_from_env from gooddata_eval.core.granularity import canonical_date_uri from gooddata_eval.core.models import AacBucketRef, AacQueryField, CreatedVisualization @@ -56,6 +57,16 @@ class FilterScores: date_ok: bool ranking_ok: bool attribute_ok: bool + # Set only when `date_ok` holds because the actual date filter is the one the dashboard + # in the item's `user_context` applies (see `check_filters`), not because it matched the + # expected output: the date dataset (`using`) of that filter. The dashboard filter may name + # none, so any date dataset is accepted and the one the agent picked is recorded here. + date_inherited_dataset: str | None = None + + @property + def date_inherited(self) -> bool: + """Whether `date_ok` holds only through the dashboard-context rule.""" + return self.date_inherited_dataset is not None @property def all_ok(self) -> bool: @@ -503,18 +514,93 @@ def normalized_filters(viz: CreatedVisualization, today: date | None = None) -> return {"date": sorted(date_set), "ranking": sorted(ranking_set), "attribute": sorted(attr_set)} +def dashboard_date_filters(user_context: object) -> list[dict]: + """The date filters of the dashboard open in ``user_context`` (``view.dashboard.filters``). + + Anything that is not shaped like that -- no context, no dashboard, a non-list ``filters``, + an entry that is not a ``date_filter`` object -- contributes nothing rather than raising: + the context is the chat API's schema and is not validated by the dataset loader. + """ + if not isinstance(user_context, Mapping): + return [] + view = user_context.get("view") + dashboard = view.get("dashboard") if isinstance(view, Mapping) else None + filters = dashboard.get("filters") if isinstance(dashboard, Mapping) else None + if not isinstance(filters, list): + return [] + return [f for f in filters if isinstance(f, dict) and f.get("type") == "date_filter"] + + +def _inherited_date_filter_dataset( + exp_date: set[str], + act_date: set[str], + actual: CreatedVisualization, + dashboard_filters: list[dict], + today: date, +) -> str | None: + """The dataset of the actual date filter when it is the dashboard's own, else None. + + Holds only when the expected output states no date filter and the actual chart has + exactly one, whose span equals that of one of the dashboard's date filters resolved + against the same ``today``. The dashboard's common date filter names no dataset (it + resolves per widget), so it is compared on the actual filter's dataset and accepts any + date dataset. A dashboard date filter that names one in ``using`` accepts only that one. + """ + if exp_date or not dashboard_filters: + return None + actual_filters = [f for f in actual.query.filter_by.values() if f.get("type") == "date_filter"] + if len(actual_filters) != 1: + return None + dataset = actual_filters[0].get("using", "") + fields = actual.query.fields + for dashboard_filter in dashboard_filters: + candidate = _normalize_date_filter( + {**dashboard_filter, "using": dashboard_filter.get("using") or dataset}, fields, today + ) + if json.dumps(candidate, sort_keys=True) in act_date: + return dataset if isinstance(dataset, str) else "" + return None + + def check_filters( - expected: CreatedVisualization, actual: CreatedVisualization, today: date | None = None + expected: CreatedVisualization, + actual: CreatedVisualization, + today: date | None = None, + *, + user_context: Mapping | None = None, + accept_context_date_filter: bool | None = None, ) -> FilterScores: + """Compare the filters of two visualizations, per category. + + Every category compares exactly. One opt-in exception, for the date category only: + with ``accept_context_date_filter`` (None reads ``GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER``, + off by default), an actual date filter equal to a date filter of the dashboard in + ``user_context`` is accepted when the expected output has none -- the agent carried the + open dashboard's period into the new chart, which the product accepts. The result then + has ``date_inherited`` set. Without the flag, or without a dashboard date filter in + ``user_context``, scoring is exactly the strict comparison. + """ # One anchor for both sides: resolving each against its own date.today() would # score inconsistently for a run that straddles midnight. today = today or date.today() exp_date, exp_rank, exp_attr = _split_and_normalize_filters(expected, today) act_date, act_rank, act_attr = _split_and_normalize_filters(actual, today) + date_ok = act_date == exp_date + inherited_dataset: str | None = None + if not date_ok and user_context is not None: + enabled = ( + _accept_context_date_filter_from_env() if accept_context_date_filter is None else accept_context_date_filter + ) + if enabled: + inherited_dataset = _inherited_date_filter_dataset( + exp_date, act_date, actual, dashboard_date_filters(user_context), today + ) + date_ok = inherited_dataset is not None return FilterScores( - date_ok=act_date == exp_date, + date_ok=date_ok, ranking_ok=act_rank == exp_rank, attribute_ok=act_attr == exp_attr, + date_inherited_dataset=inherited_dataset, ) diff --git a/packages/gooddata-eval/tests/conftest.py b/packages/gooddata-eval/tests/conftest.py index 0f77eaa8a..f40645208 100644 --- a/packages/gooddata-eval/tests/conftest.py +++ b/packages/gooddata-eval/tests/conftest.py @@ -26,6 +26,7 @@ def fixtures_dir() -> Path: "GOODDATA_EVAL_CHAT_TURN_TIMEOUT_S", "GOODDATA_EVAL_CHAT_ITEM_TIMEOUT_S", "GOODDATA_EVAL_JOIN_GENAI_TRACE", + "GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER", ) diff --git a/packages/gooddata-eval/tests/test_agentic_alert_skill.py b/packages/gooddata-eval/tests/test_agentic_alert_skill.py index 09cfab387..9513102e3 100644 --- a/packages/gooddata-eval/tests/test_agentic_alert_skill.py +++ b/packages/gooddata-eval/tests/test_agentic_alert_skill.py @@ -26,7 +26,7 @@ render_turn_for_simulated_user, run_agentic_alert_skill, ) -from gooddata_eval.core.chat.sse_client import ChatError +from gooddata_eval.core.chat.sse_client import ChatError, TurnIncompleteError, TurnTimeoutError from gooddata_eval.core.models import ChatResult, LoopExit _DATE_FILTER = { @@ -1932,3 +1932,21 @@ def test_an_alert_created_before_the_stream_broke_is_still_cleaned_up(): assert mock_delete.call_count == 1 assert mock_delete.call_args.args[-1] == "alert-1" + + +@pytest.mark.parametrize( + ("exc", "exit_value"), + [ + (TurnTimeoutError("agent exceeded the 60s turn budget"), "agent_timeout"), + (TurnIncompleteError("no final answer", reason="max iterations"), "turn_incomplete"), + ], +) +def test_exit_reason_names_the_agents_own_chat_failure(exc, exit_value): + mock_client = MagicMock() + mock_client.create_conversation.return_value = "conv-1" + mock_client.send_message.side_effect = exc + + detail = _run_alert(mock_client, max_iterations=3) + + assert detail["exit_reason"] == exit_value + assert detail["alert_created"] is False diff --git a/packages/gooddata-eval/tests/test_agentic_anomaly_detection.py b/packages/gooddata-eval/tests/test_agentic_anomaly_detection.py index 5c1e25d8b..4bdfe156b 100644 --- a/packages/gooddata-eval/tests/test_agentic_anomaly_detection.py +++ b/packages/gooddata-eval/tests/test_agentic_anomaly_detection.py @@ -16,6 +16,7 @@ evaluate_agentic_anomaly_detection, run_agentic_anomaly_detection, ) +from gooddata_eval.core.chat.sse_client import TurnIncompleteError, TurnTimeoutError from gooddata_eval.core.models import ChatResult, LoopExit _MODULE = "gooddata_eval.core.agentic.anomaly_detection" @@ -431,7 +432,7 @@ def trace(self, _conversation_id): def observe(self, _trace, _run_idx, *, conversation_id=None): yield "trace-id" - def score(self, _tid, *, name, value, data_type): + def score(self, _tid, *, name, value, data_type, comment=None): self.scores[name] = value def quality(self, _tid, *, strict_checks, latency_sec, cost_usd): @@ -539,3 +540,15 @@ def test_any_gate_still_passes_the_same_item(): def test_the_default_gate_is_pass_at_k(): assert _two_runs().runs_passed == 1 + + +@pytest.mark.parametrize( + ("exc", "loop_exit"), + [ + (TurnTimeoutError("agent exceeded the 60s turn budget"), LoopExit.AGENT_TIMEOUT), + (TurnIncompleteError("no final answer", reason="max iterations"), LoopExit.TURN_INCOMPLETE), + ], +) +def test_the_agents_own_chat_failure_is_recorded_as_its_own_exit(exc, loop_exit): + summary = _run([_chat(_pair()), exc], k=2) + assert summary.run_results[1].exit_reason is loop_exit diff --git a/packages/gooddata-eval/tests/test_agentic_conversation.py b/packages/gooddata-eval/tests/test_agentic_conversation.py index ca0f2dc2d..f08b0d63f 100644 --- a/packages/gooddata-eval/tests/test_agentic_conversation.py +++ b/packages/gooddata-eval/tests/test_agentic_conversation.py @@ -53,7 +53,7 @@ run_agentic_conversation, ) from gooddata_eval.core.chat.render import render_answer_text -from gooddata_eval.core.chat.sse_client import ChatError, TurnIncompleteError +from gooddata_eval.core.chat.sse_client import ChatError, TurnIncompleteError, TurnTimeoutError from gooddata_eval.core.chat.sse_client import ChatError as ChatError_ from gooddata_eval.core.evaluators._llm_judge import JudgeResponseError from gooddata_eval.core.models import ChatResult, LoopExit, ToolCallEvent @@ -1209,6 +1209,32 @@ def test_a_partial_result_joins_the_shared_timeline_like_any_other_turn(): assert result.total_steps == 5 +def test_an_agent_timeout_ends_its_turn_with_its_own_exit_and_still_reads_as_an_error(): + mock_client = MagicMock() + mock_client.create_conversation.return_value = "conv-1" + mock_client.send_message.side_effect = [ + _metric_turn_result([_skills_tc("metric"), _create_metric_tc("m1")]), + TurnTimeoutError("agent exceeded the 60s turn budget"), + ] + + with ( + patch("gooddata_eval.core.agentic.conversation.ChatClient", return_value=mock_client), + patch("gooddata_eval.core.agentic.conversation.GoodDataSdk"), + ): + result = run_agentic_conversation( + host="http://host/api/v1/actions/workspaces/ws1/ai", + token="tok", + workspace_id="ws1", + fixture=_two_metric_turn_fixture(), + ) + + assert result.turn_results[0].exit_reason is LoopExit.SUCCESS + assert result.turn_results[1].exit_reason is LoopExit.AGENT_TIMEOUT + # The turn's no_error reads exactly as it did when a timeout was recorded as CHAT_ERROR. + assert result.turn_results[1].no_error is False + assert result.turn_results[1].output_present is False + + def test_a_non_chat_exception_still_propagates(): """Only chat faults are absorbed. A programming error must not be relabelled CHAT_ERROR.""" mock_client = MagicMock() diff --git a/packages/gooddata-eval/tests/test_agentic_dashboard_skill.py b/packages/gooddata-eval/tests/test_agentic_dashboard_skill.py index 9b6075b9f..d053be358 100644 --- a/packages/gooddata-eval/tests/test_agentic_dashboard_skill.py +++ b/packages/gooddata-eval/tests/test_agentic_dashboard_skill.py @@ -20,7 +20,7 @@ evaluate_dashboard_response, run_agentic_dashboard_skill, ) -from gooddata_eval.core.chat.sse_client import ChatError +from gooddata_eval.core.chat.sse_client import ChatError, TurnIncompleteError, TurnTimeoutError from gooddata_eval.core.models import ChatResult, LoopExit, ToolCallEvent # Ids and titles are the ones the eval layout seeds; shapes are trimmed from real runs of @@ -2947,6 +2947,22 @@ def test_a_caller_supplied_conversation_is_not_deleted(self): client.create_conversation.assert_not_called() client.delete_conversation.assert_not_called() + @pytest.mark.parametrize( + ("exc", "loop_exit"), + [ + (TurnTimeoutError("agent exceeded the 60s turn budget"), LoopExit.AGENT_TIMEOUT), + (TurnIncompleteError("no final answer", reason="max iterations"), LoopExit.TURN_INCOMPLETE), + ], + ) + def test_the_agents_own_chat_failure_is_recorded_as_its_own_exit(self, exc, loop_exit) -> None: + client = MagicMock() + client.send_message.side_effect = exc + + summary = _run_with(client, _DC05_EXPECTED) + + assert summary.run_results[0].exit_reason is loop_exit + assert summary.pass_at_k is False + def test_a_chat_error_is_recorded_rather_than_left_to_escape(self) -> None: """It used to escape run_agentic_dashboard_skill, taking every completed K-run with it: no DashboardRunResult, no exit_reason, just a bare error on the item. Recorded diff --git a/packages/gooddata-eval/tests/test_agentic_failure_message.py b/packages/gooddata-eval/tests/test_agentic_failure_message.py new file mode 100644 index 000000000..0d60acc20 --- /dev/null +++ b/packages/gooddata-eval/tests/test_agentic_failure_message.py @@ -0,0 +1,356 @@ +# (C) 2026 GoodData Corporation +"""The failing-runs head of an agentic failure message, and the per-run records it reads.""" + +import importlib +import inspect +from dataclasses import dataclass, field +from unittest.mock import MagicMock, patch + +import pytest +from gooddata_eval.core.agentic import metric_skill as metric_skill_mod +from gooddata_eval.core.agentic import visualization as agentic_visualization +from gooddata_eval.core.agentic._failed_runs import build_failed_runs +from gooddata_eval.core.agentic._failure_message import ( + FailedCheck, + failed_check_records, + failing_runs_note, + failure_head, +) +from gooddata_eval.core.agentic.metric_skill import MetricSkillAssertionError, evaluate_agentic_metric_skill +from gooddata_eval.core.agentic.visualization import VisualizationAssertionError, evaluate_agentic_visualization +from gooddata_eval.core.models import ChatResult, CreatedVisualization, LoopExit +from gooddata_eval.core.outcome import classify_run + +# ── the renderer ───────────────────────────────────────────────────────────── + + +def test_a_failed_check_records_only_what_the_kind_knows() -> None: + assert FailedCheck("metric_correct").as_record() == {"name": "metric_correct"} + assert FailedCheck("filters_correct", label="date", expected=[], actual=["MONTH -11..0"]).as_record() == { + "name": "filters_correct", + "label": "date", + "expected": [], + "actual": ["MONTH -11..0"], + } + # None is a real expected value, not "unknown". + assert FailedCheck("x", expected=None).as_record() == {"name": "x", "expected": None} + + +def test_the_note_lists_each_failing_run_with_its_exit_and_failing_checks_only() -> None: + failed_runs = [ + { + "run_index": 2, + "exit_reason": "success", + "failed_checks": failed_check_records( + [FailedCheck("filters_correct", label="date", expected=[], actual="MONTH -11..0")] + ), + }, + {"run_index": 3, "exit_reason": "budget_exhausted", "failed_checks": [{"name": "metric_correct"}]}, + ] + + note = failing_runs_note(failed_runs, 3) + + assert note == ( + "run 2/3 FAIL (loop exit: success): filters_correct[date: expected none, actual MONTH -11..0]; " + "run 3/3 FAIL (loop exit: budget_exhausted): metric_correct" + ) + + +def test_the_head_starts_with_the_gate_note() -> None: + gate_note = "Gate pass^3 failed: 2/3 runs passed — unstable, not a clean failure." + failed_runs = [{"run_index": 2, "exit_reason": "success", "failed_checks": [{"name": "metric_correct"}]}] + + head = failure_head(gate_note, failed_runs, 3) + + assert head.startswith(gate_note) + assert head.endswith("Failing runs: run 2/3 FAIL (loop exit: success): metric_correct.") + + +def test_the_head_is_the_gate_note_alone_without_failing_runs() -> None: + assert failure_head("Gate pass@2 failed: 0/2 runs passed.", [], 2) == "Gate pass@2 failed: 0/2 runs passed." + + +def test_a_run_without_recorded_checks_says_so_and_a_kind_without_the_key_says_nothing() -> None: + assert failing_runs_note([{"run_index": 1, "exit_reason": None, "failed_checks": []}], 1) == ( + "run 1/1 FAIL: no failing check recorded" + ) + assert failing_runs_note([{"run_index": 1, "exit_reason": "agent_silent"}], 1) == ( + "run 1/1 FAIL (loop exit: agent_silent)" + ) + + +def test_an_errored_run_is_labelled_with_its_reason_and_lists_no_checks() -> None: + record = { + "run_index": 1, + "exit_reason": "simulated_user_failed", + "outcome": {"status": "error", "reason": "simulated_user_failed"}, + "failed_checks": [{"name": "metric_created"}], + } + assert failing_runs_note([record], 3) == "run 1/3 ERROR (simulated_user_failed)" + + +def test_a_long_value_is_cut_to_one_short_line() -> None: + record = {"run_index": 1, "failed_checks": [{"name": "maql_correct", "actual": "SELECT\n" + "x" * 500}]} + note = failing_runs_note([record], 1) + assert "\n" not in note + assert note.endswith("...]") + assert len(note) < 250 + + +# ── the records ────────────────────────────────────────────────────────────── + + +@dataclass +class _Run: + ok: bool + exit_reason: LoopExit = LoopExit.SUCCESS + conversation_id: str = "c" + tool_call_events: list = field(default_factory=list) + + +def test_every_failing_record_carries_its_exit_reason_and_failed_checks() -> None: + runs = [_Run(True), _Run(False, LoopExit.BUDGET_EXHAUSTED)] + + records = build_failed_runs( + runs, + passed=lambda r: r.ok, + detail=lambda r: {}, + failed_checks=lambda r: [{"name": "metric_correct"}], + ) + + assert len(records) == 1 + assert records[0]["run_index"] == 2 + assert records[0]["exit_reason"] == "budget_exhausted" + assert records[0]["failed_checks"] == [{"name": "metric_correct"}] + + +def test_a_kind_that_names_no_checks_writes_no_failed_checks_key() -> None: + records = build_failed_runs([object()], passed=lambda r: False, detail=lambda r: {}) + assert records[0]["exit_reason"] is None + assert "failed_checks" not in records[0] + + +@dataclass +class _StrRun: + exit_reason: str = "chat_error" + + +def test_a_string_exit_reason_gets_the_same_outcome_as_its_enum() -> None: + # exit_reason and outcome must agree whether a kind records the member or its value. + for exit_reason in (LoopExit.SIMULATED_USER_FAILED, "simulated_user_failed"): + record = build_failed_runs([_StrRun(exit_reason)], passed=lambda r: False, detail=lambda r: {})[0] + assert record["exit_reason"] == "simulated_user_failed" + assert record["outcome"] == {"status": "error", "reason": "simulated_user_failed"} + + +@pytest.mark.parametrize( + "exit_reason", [LoopExit.CHAT_ERROR, "chat_error", "budget_exhausted"], ids=["member", "value", "agent_exit"] +) +def test_a_chat_error_or_agent_exit_is_recorded_and_rendered_as_a_failure(exit_reason) -> None: + records = build_failed_runs( + [_StrRun(exit_reason)], passed=lambda r: False, detail=lambda r: {}, failed_checks=lambda r: [] + ) + + value = LoopExit(exit_reason).value + assert records[0]["exit_reason"] == value + assert records[0]["outcome"] == {"status": "fail", "reason": value} + assert failing_runs_note(records, 3) == f"run 1/3 FAIL (loop exit: {value}): no failing check recorded" + + +def test_a_simulated_user_failure_is_recorded_and_rendered_as_an_error() -> None: + records = build_failed_runs( + [_StrRun("simulated_user_failed")], passed=lambda r: False, detail=lambda r: {}, failed_checks=lambda r: [] + ) + + assert records[0]["outcome"]["status"] == "error" + assert failing_runs_note(records, 3) == "run 1/3 ERROR (simulated_user_failed)" + + +@dataclass +class _JudgedRun: + judge_error: str | None + exit_reason: str = "success" + + +def test_a_run_the_judge_could_not_grade_is_recorded_as_an_error() -> None: + (record,) = build_failed_runs([_JudgedRun("empty body twice")], passed=lambda r: False, detail=lambda r: {}) + assert record["outcome"] == {"status": "error", "reason": "judge_failed"} + + +def test_a_failed_check_record_keeps_short_values_and_caps_long_ones() -> None: + long_actual = {"filters": ["x" * 400]} + records = build_failed_runs( + [_StrRun("budget_exhausted")], + passed=lambda r: False, + detail=lambda r: {"actual": long_actual}, + failed_checks=lambda r: [ + {"name": "metrics_correct", "expected": ["metric/a"], "actual": ["metric/b"]}, + {"name": "filters_correct", "label": "attribute", "expected": [], "actual": long_actual}, + ], + ) + + short, long_check = records[0]["failed_checks"] + assert short == {"name": "metrics_correct", "expected": ["metric/a"], "actual": ["metric/b"]} + assert long_check["expected"] == [] + assert isinstance(long_check["actual"], str) + assert len(long_check["actual"]) == 160 + assert long_check["actual"].endswith("...") + # The run's detail keeps the whole value. + assert records[0]["detail"]["actual"] == long_actual + + +def test_a_fail_run_names_its_loop_exit_as_its_reason() -> None: + assert classify_run(LoopExit.SUCCESS, passed=False).reason == "success" + + +def test_a_metric_layout_failure_names_the_layout_problem_not_two_equal_sets() -> None: + ev = MagicMock( + scored_without_chart=False, + visualization_created=True, + cross_ref_valid=True, + metrics_correct=False, + expected_metric_uris={"metric/revenue"}, + actual_metric_uris={"metric/revenue"}, + metric_layout_problems=["metric/revenue is on the secondary axis"], + dimensions_correct=True, + filters_correct=True, + sorts_correct=True, + viz_type_hard=True, + requires_execution=False, + buckets_correct=None, + answer_correct=None, + format_correct=None, + part_present=None, + ) + run = MagicMock(eval_result=ev) + + assert agentic_visualization._failed_checks(run) == [ + {"name": "metrics_correct", "label": "layout", "actual": ["metric/revenue is on the secondary axis"]} + ] + + +# ── visualization: a partial gate failure names the run that failed ────────── + + +def _viz(filter_by: dict | None = None) -> dict: + return { + "id": "v1", + "type": "column_chart", + "query": { + "fields": {"m": {"using": "metric/revenue"}, "d": {"using": "label/date.quarter"}}, + "filter_by": filter_by or {}, + }, + "metrics": ["m"], + "view_by": ["d"], + } + + +def _viz_turn(filter_by: dict | None = None) -> ChatResult: + return ChatResult.model_validate( + {"createdVisualizations": {"objects": [_viz(filter_by)], "reasoning": ""}, "toolCallEvents": []} + ) + + +_LAST_YEAR = {"date": {"type": "date_filter", "using": "dataset/date", "granularity": "MONTH", "from": -11, "to": 0}} + + +def test_a_partial_visualization_failure_lists_the_failing_run_and_keeps_the_best_run_summary() -> None: + client = MagicMock() + client.create_conversation.side_effect = ["conv-1", "conv-2", "conv-3"] + client.send_message.side_effect = [_viz_turn(), _viz_turn(_LAST_YEAR), _viz_turn()] + + with ( + patch.object(agentic_visualization, "ChatClient", return_value=client), + pytest.raises(VisualizationAssertionError) as exc_info, + ): + evaluate_agentic_visualization( + host="https://example.com", + token="tok", + workspace_id="ws", + question="Show revenue by quarter", + expected_outputs=[CreatedVisualization.model_validate(_viz())], + k=3, + gate="power", + ) + + message = str(exc_info.value) + lines = message.splitlines() + gate_line = next(line for line in lines if line.startswith("Gate pass^3 failed")) + assert gate_line.startswith("Gate pass^3 failed: 2/3 runs passed — unstable, not a clean failure. Failing runs: ") + assert "run 2/3 FAIL (loop exit: success): filters_correct[date: expected none, actual " in gate_line + # Only the failing checks of the failing run, and no passing run. + assert "metrics_correct" not in gate_line + assert "run 1/3" not in gate_line + assert "run 3/3" not in gate_line + # The best-run summary that followed the gate note is still there, unchanged. + assert lines[1] == "Agentic Visualization Assertion Failed! (Critical Mode)" + assert "Strict Check Summary:" in message + assert " Filters Correct : True\n" in message + + (record,) = exc_info.value.failed_runs + assert record["run_index"] == 2 + assert record["exit_reason"] == "success" + assert record["detail"]["exit_reason"] == "success" + assert [c["name"] for c in record["failed_checks"]] == ["filters_correct"] + assert record["failed_checks"][0]["label"] == "date" + assert record["failed_checks"][0]["expected"] == [] + + +# ── metric skill: same head, expected/actual MAQL of the failing run ───────── + + +def _metric_turn(maql: str) -> ChatResult: + return ChatResult.model_validate( + { + "textResponse": "done", + "toolCallEvents": [ + { + "functionName": "create_metric", + "functionArguments": "{}", + "result": f'{{"data": {{"maql": "{maql}"}}}}', + } + ], + } + ) + + +def test_a_partial_metric_failure_lists_the_failing_run_with_its_maql() -> None: + client = MagicMock() + client.create_conversation.return_value = "conv-1" + client.send_message.side_effect = [_metric_turn("SELECT {metric/foo}"), _metric_turn("SELECT {metric/bar}")] + + with ( + patch.object(metric_skill_mod, "ChatClient", return_value=client), + pytest.raises(MetricSkillAssertionError) as exc_info, + ): + evaluate_agentic_metric_skill( + host="http://host/api/v1/actions/workspaces/ws1/ai", + token="tok", + workspace_id="ws1", + question="Create metric foo", + expected_output={"maql": "SELECT {metric/foo}"}, + k=2, + max_iterations=1, + gate="power", + ) + + message = str(exc_info.value) + assert message.startswith( + "Metric skill assertion failed. Gate pass^2 failed: 1/2 runs passed — unstable, not a clean failure. " + "Failing runs: run 2/2 FAIL (loop exit: success): " + 'maql_correct[expected ["SELECT {metric/foo}"], actual SELECT {metric/bar}].' + ) + # The best run's summary follows, as before. + assert "metric_created=True, maql_correct=True. Expected MAQL (candidates): 'SELECT {metric/foo}'." in message + (record,) = exc_info.value.failed_runs + assert record["exit_reason"] == "success" + assert record["failed_checks"] == [ + {"name": "maql_correct", "expected": ["SELECT {metric/foo}"], "actual": "SELECT {metric/bar}"} + ] + + +@pytest.mark.parametrize("module_name", ["visualization", "alert_skill", "metric_skill", "dashboard_skill"]) +def test_each_checked_kind_heads_its_message_through_the_shared_renderer(module_name: str) -> None: + source = inspect.getsource(importlib.import_module(f"gooddata_eval.core.agentic.{module_name}")) + assert "gate_head(gate, run_labels, runs_passed, failed_runs)" in source + assert "failed_checks=_failed_checks" in source diff --git a/packages/gooddata-eval/tests/test_agentic_kda_skill.py b/packages/gooddata-eval/tests/test_agentic_kda_skill.py index 6a1690ebb..8f13a4093 100644 --- a/packages/gooddata-eval/tests/test_agentic_kda_skill.py +++ b/packages/gooddata-eval/tests/test_agentic_kda_skill.py @@ -16,8 +16,8 @@ evaluate_agentic_kda_skill, run_agentic_kda_skill, ) -from gooddata_eval.core.chat.sse_client import ChatError, TransientChatError -from gooddata_eval.core.models import ChatResult +from gooddata_eval.core.chat.sse_client import ChatError, TransientChatError, TurnIncompleteError, TurnTimeoutError +from gooddata_eval.core.models import ChatResult, LoopExit _EXPECTED = {"Measure": {"type": "metric", "id": "revenue"}} @@ -1296,3 +1296,29 @@ def _capture(_submit, _identity, **kwargs): assert scores["turns"] == 1 assert scores["steps"] == 1 + + +@pytest.mark.parametrize( + ("exc", "loop_exit"), + [ + (TurnTimeoutError("agent exceeded the 60s turn budget"), LoopExit.AGENT_TIMEOUT), + (TurnIncompleteError("no final answer", reason="max iterations"), LoopExit.TURN_INCOMPLETE), + (TransientChatError("gen-ai returned 503"), LoopExit.CHAT_ERROR), + ], +) +def test_run_agentic_kda_skill_names_why_the_chat_call_failed(exc, loop_exit): + mock_client = _client() + mock_client.send_message.side_effect = exc + + with _patched(mock_client): + summary = run_agentic_kda_skill( + host="http://host/api/v1/actions/workspaces/ws1/ai", + token="tok", + workspace_id="ws1", + question="What drove revenue change?", + expected_output=_EXPECTED, + k=1, + max_iterations=1, + ) + + assert summary.best.exit_reason is loop_exit diff --git a/packages/gooddata-eval/tests/test_agentic_metric_skill.py b/packages/gooddata-eval/tests/test_agentic_metric_skill.py index 5efcbb0bb..b8ebb0c63 100644 --- a/packages/gooddata-eval/tests/test_agentic_metric_skill.py +++ b/packages/gooddata-eval/tests/test_agentic_metric_skill.py @@ -22,7 +22,7 @@ generate_simulated_response, run_agentic_metric_skill, ) -from gooddata_eval.core.chat.sse_client import ChatError +from gooddata_eval.core.chat.sse_client import ChatError, TurnIncompleteError, TurnTimeoutError from gooddata_eval.core.models import ChatResult, LoopExit, ToolCallEvent from gooddata_eval.core.timing import TIMERS_ENV_VAR @@ -1098,3 +1098,81 @@ def call(mid: str, created_new: bool) -> ToolCallEvent: ) assert _extract_created_metric_ids([call("new", True), call("old", False)]) == ["new"] + + +@pytest.mark.parametrize( + ("exc", "loop_exit"), + [ + (TurnTimeoutError("agent exceeded the 60s turn budget"), LoopExit.AGENT_TIMEOUT), + (TurnIncompleteError("no final answer", reason="max iterations"), LoopExit.TURN_INCOMPLETE), + ], +) +def test_the_agents_own_chat_failure_is_recorded_as_its_own_exit(exc, loop_exit): + mock_client = _client() + mock_client.send_message.side_effect = exc + + with _patched(mock_client): + summary = run_agentic_metric_skill( + host="http://host/api/v1/actions/workspaces/ws1/ai", + token="tok", + workspace_id="ws1", + question="Create metric foo", + expected_output={"maql": "SELECT {metric/foo}"}, + k=1, + max_iterations=3, + ) + + assert summary.run_results[0].exit_reason is loop_exit + assert summary.pass_at_k is False + + +def test_a_chat_error_run_is_a_failure_scored_as_before(): + mock_client = _client() + mock_client.send_message.side_effect = [ + ChatError("gen-ai fell over"), + ChatResult.model_validate( + { + "textResponse": "done", + "toolCallEvents": [ + { + "functionName": "create_metric", + "functionArguments": "{}", + "result": '{"data": {"maql": "SELECT {metric/foo}"}}', + } + ], + } + ), + ] + captured = {} + + def _capture(_submit, _identity, **kwargs): + captured["write_scores"] = kwargs["write_scores"] + + with ( + _patched(mock_client), + patch("gooddata_eval.core.agentic.metric_skill.submit_trace_scoring", _capture), + ): + evaluate_agentic_metric_skill( + host="http://host/api/v1/actions/workspaces/ws1/ai", + token="tok", + workspace_id="ws1", + question="Create metric foo", + expected_output={"maql": "SELECT {metric/foo}"}, + k=2, + max_iterations=1, + langfuse=MagicMock(), + dataset_item_id="item-1", + ) + + ctx = MagicMock() + run_ids = [MagicMock(name="run-1"), MagicMock(name="run-2")] + ctx.observe.return_value.__enter__.side_effect = run_ids + captured["write_scores"](ctx) + by_run = {id(t): [c.kwargs["name"] for c in ctx.score.call_args_list if c.args[0] is t] for t in run_ids} + + first, second = by_run[id(run_ids[0])], by_run[id(run_ids[1])] + # A chat API error is on the product path: a FAIL, with no harness marker. + assert "eval_error" not in first + assert "gate_passed" in first + assert "eval_error" not in second + assert "gate_passed" in second diff --git a/packages/gooddata-eval/tests/test_agentic_observe_experiment.py b/packages/gooddata-eval/tests/test_agentic_observe_experiment.py index 83f31a277..44a4d4fba 100644 --- a/packages/gooddata-eval/tests/test_agentic_observe_experiment.py +++ b/packages/gooddata-eval/tests/test_agentic_observe_experiment.py @@ -345,3 +345,15 @@ def test_the_run_context_forwards_its_window_and_item_input_to_observe(): "item_input": "Show revenue by month", "output": {"passed": True}, } + + +def test_the_run_context_forwards_a_score_comment_only_when_given(): + lf = MagicMock() + ctx = RunTraceContext({}, lf, "client", "item-1", "base", True, {}, _WINDOW, "q") + + ctx.score("tid", name="gate_passed", value=True, data_type="BOOLEAN") + ctx.score("tid", name="eval_error", value=1, data_type="NUMERIC", comment="backend: the chat call failed") + + plain, commented = lf.score_safe.call_args_list + assert plain.kwargs == {"name": "gate_passed", "value": True, "data_type": "BOOLEAN"} + assert commented.kwargs["comment"] == "backend: the chat call failed" diff --git a/packages/gooddata-eval/tests/test_agentic_visualization.py b/packages/gooddata-eval/tests/test_agentic_visualization.py index 431c3f329..c50d2fddb 100644 --- a/packages/gooddata-eval/tests/test_agentic_visualization.py +++ b/packages/gooddata-eval/tests/test_agentic_visualization.py @@ -18,7 +18,7 @@ evaluate_agentic_visualization, run_agentic_visualization, ) -from gooddata_eval.core.chat.sse_client import ChatError +from gooddata_eval.core.chat.sse_client import ChatError, TurnIncompleteError, TurnTimeoutError from gooddata_eval.core.evaluators.visualization import ExpectationOptions from gooddata_eval.core.models import ChatResult, CreatedVisualization, LoopExit @@ -822,3 +822,35 @@ def test_a_failed_request_with_a_cut_off_answer_fails_an_item_without_a_chart() assert result.exit_reason is LoopExit.CHAT_ERROR assert result.eval_result.strict_pass is False assert result.eval_result.answer_problems == ["the chat request failed before the answer ended"] + + +_AGENT_CHAT_FAILURES = [ + (TurnTimeoutError("agent exceeded the 60s turn budget"), LoopExit.AGENT_TIMEOUT), + (TurnIncompleteError("no final answer", reason="max iterations"), LoopExit.TURN_INCOMPLETE), +] + + +@pytest.mark.parametrize(("exc", "loop_exit"), _AGENT_CHAT_FAILURES) +def test_execute_single_run_records_the_agents_own_failure_on_the_opening_request(exc, loop_exit): + client = MagicMock() + client.send_message.side_effect = exc + + result = _execute_single_run(client, "conv-1", "Show revenue", [_expected()]) + + assert result.exit_reason is loop_exit + assert result.eval_result.visualization_created is False + # Like a chat error, the loop does not run after the opening failure. + assert client.send_message.call_count == 1 + + +@pytest.mark.parametrize(("exc", "loop_exit"), _AGENT_CHAT_FAILURES) +def test_execute_single_run_records_the_agents_own_failure_on_a_follow_up(monkeypatch, exc, loop_exit): + monkeypatch.setattr( + "gooddata_eval.core.agentic.visualization.generate_simulated_response", lambda *a, **k: "the revenue one" + ) + client = MagicMock() + client.send_message.side_effect = [_chat_clarification(), exc] + + result = _execute_single_run(client, "conv-1", "Show revenue", [_expected()]) + + assert result.exit_reason is loop_exit diff --git a/packages/gooddata-eval/tests/test_agentic_what_if.py b/packages/gooddata-eval/tests/test_agentic_what_if.py index 38b04ebf3..4ee0e4ab0 100644 --- a/packages/gooddata-eval/tests/test_agentic_what_if.py +++ b/packages/gooddata-eval/tests/test_agentic_what_if.py @@ -13,6 +13,7 @@ evaluate_agentic_what_if, run_agentic_what_if, ) +from gooddata_eval.core.chat.sse_client import TurnIncompleteError, TurnTimeoutError from gooddata_eval.core.models import ChatResult, LoopExit _MODULE = "gooddata_eval.core.agentic.what_if" @@ -385,7 +386,7 @@ def trace(self, _conversation_id): def observe(self, _trace, _run_idx, *, conversation_id=None): yield "trace-id" - def score(self, _tid, *, name, value, data_type): + def score(self, _tid, *, name, value, data_type, comment=None): self.scores[name] = value def quality(self, _tid, *, strict_checks, latency_sec, cost_usd): @@ -499,3 +500,15 @@ def test_any_gate_still_passes_the_same_item(): def test_the_default_gate_is_pass_at_k(): assert _two_runs().runs_passed == 1 + + +@pytest.mark.parametrize( + ("exc", "loop_exit"), + [ + (TurnTimeoutError("agent exceeded the 60s turn budget"), LoopExit.AGENT_TIMEOUT), + (TurnIncompleteError("no final answer", reason="max iterations"), LoopExit.TURN_INCOMPLETE), + ], +) +def test_the_agents_own_chat_failure_is_recorded_as_its_own_exit(exc, loop_exit): + summary = _run([_chat(_pair()), exc], k=2) + assert summary.run_results[1].exit_reason is loop_exit diff --git a/packages/gooddata-eval/tests/test_cli.py b/packages/gooddata-eval/tests/test_cli.py index ffc7404a8..3987f98e3 100644 --- a/packages/gooddata-eval/tests/test_cli.py +++ b/packages/gooddata-eval/tests/test_cli.py @@ -1,7 +1,7 @@ # (C) 2026 GoodData Corporation import io from concurrent.futures import ThreadPoolExecutor, as_completed -from datetime import UTC, datetime +from datetime import UTC, date, datetime from unittest.mock import MagicMock import httpx @@ -9,13 +9,20 @@ import pytest from gooddata_eval.cli import main as cli_main from gooddata_eval.cli.main import _parse_model_arg -from gooddata_eval.core.config import JUDGE_MODEL_ENV_VAR, RunConfig, judge_model +from gooddata_eval.core.config import ( + ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, + JUDGE_MODEL_ENV_VAR, + RunConfig, + accept_context_date_filter, + judge_model, +) from gooddata_eval.core.connection import ( ConnectionError_, # noqa: F401 - used in test_cli_operational_error_exits_nonzero ) from gooddata_eval.core.dataset.langfuse_source import _item_from_raw -from gooddata_eval.core.models import DatasetItem +from gooddata_eval.core.models import CreatedVisualization, DatasetItem from gooddata_eval.core.runner import EvalReport, ItemReport +from gooddata_eval.core.scoring import FilterScores, check_filters from gooddata_eval.core.timing import TIMERS_ENV_VAR, timers_enabled from gooddata_eval.core.workspace import ActiveLlmProvider, ResolvedModel from rich.console import Console @@ -1197,3 +1204,48 @@ def test_routing_backend_closes_every_client() -> None: for client in (chat, summary, takeaway): client.close.assert_called_once_with() + + +def test_accept_context_date_filter_flag_is_off_by_default_and_publishes_the_env_var(monkeypatch): + base = ["run", "--host", "h", "--workspace", "w", "--dataset", "d"] + assert cli_main.parse_args(base).accept_context_date_filter is False + assert cli_main.parse_args([*base, "--accept-context-date-filter"]).accept_context_date_filter is True + + # Set, not deleted, so monkeypatch restores what the flag writes. + monkeypatch.setenv(ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, "0") + cli_main._apply_accept_context_date_filter_flag(False) + assert accept_context_date_filter() is False + cli_main._apply_accept_context_date_filter_flag(True) + assert accept_context_date_filter() is True + + +def test_accept_context_date_filter_flag_never_clears_an_exported_env_var(monkeypatch): + monkeypatch.setenv(ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, "1") + cli_main._apply_accept_context_date_filter_flag(False) + assert accept_context_date_filter() is True + + +def test_accept_context_date_filter_flag_reaches_check_filters(monkeypatch): + def viz(filter_by: dict) -> CreatedVisualization: + return CreatedVisualization.model_validate( + { + "id": "v", + "type": "headline_chart", + "query": {"fields": {"m": {"using": "metric/m"}}, "filter_by": filter_by}, + "metrics": ["m"], + } + ) + + inherited = {"type": "date_filter", "using": "dataset/d", "granularity": "MONTH", "from": -11, "to": 0} + user_context = { + "view": {"dashboard": {"filters": [{"type": "date_filter", "granularity": "MONTH", "from": -11, "to": 0}]}} + } + + def score() -> FilterScores: + return check_filters(viz({}), viz({"f": inherited}), date(2026, 11, 12), user_context=user_context) + + monkeypatch.setenv(ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, "0") + assert score().date_ok is False # default: strict date comparison + cli_main._apply_accept_context_date_filter_flag(True) + assert score().date_ok is True + assert score().date_inherited is True diff --git a/packages/gooddata-eval/tests/test_context_date_filter.py b/packages/gooddata-eval/tests/test_context_date_filter.py new file mode 100644 index 000000000..7f9df3753 --- /dev/null +++ b/packages/gooddata-eval/tests/test_context_date_filter.py @@ -0,0 +1,400 @@ +# (C) 2026 GoodData Corporation +"""A date filter inherited from the dashboard in the item's `user_context`. + +The rule is opt-in (`GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER`, or `accept_context_date_filter=`). +With it off, every case here must score exactly as the strict comparison does. +""" + +import json +from datetime import date +from unittest.mock import MagicMock, patch + +import pytest +from gooddata_eval.core.agentic import visualization as agentic_visualization +from gooddata_eval.core.agentic.visualization import ( + VisualizationAssertionError, + evaluate_agentic_visualization, + run_agentic_visualization, +) +from gooddata_eval.core.evaluators import get_evaluator +from gooddata_eval.core.evaluators.visualization import _evaluate_against_candidates, evaluation_result_detail +from gooddata_eval.core.models import ChatResult, CreatedVisualization, DatasetItem +from gooddata_eval.core.scoring import check_filters, dashboard_date_filters + +_ENV = "GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER" +_TODAY = date(2026, 11, 12) + +# The real item from the ticket: globalmart_visualization_charts_ctx_2, run 40229016049. +_USER_CONTEXT = { + "view": { + "dashboard": { + "id": "dashboard_033", + "title": "Order Lines & Margin", + "filters": [{"type": "date_filter", "granularity": "MONTH", "from": -11, "to": 0}], + } + } +} +_INHERITED = { + "type": "date_filter", + "using": "dataset/transaction_date", + "granularity": "MONTH", + "from": -11, + "to": 0, +} + + +def _headline(filter_by: dict | None = None, metric: str = "metric/gross_margin_percent") -> dict: + return { + "id": "v", + "type": "headline_chart", + "query": {"fields": {"m": {"using": metric}}, "filter_by": filter_by or {}}, + "metrics": ["m"], + } + + +def _viz(filter_by: dict | None = None) -> CreatedVisualization: + return CreatedVisualization.model_validate(_headline(filter_by)) + + +def _date(frm: int, to: int, granularity: str = "MONTH", using: str = "dataset/transaction_date") -> dict: + return {"type": "date_filter", "using": using, "granularity": granularity, "from": frm, "to": to} + + +# ── the five acceptance-criteria cases (flag on) ───────────────────────────── + + +def test_inherited_dashboard_range_passes_and_is_marked_inherited(): + scores = check_filters( + _viz(), + _viz({"f_transaction_date": _INHERITED}), + _TODAY, + user_context=_USER_CONTEXT, + accept_context_date_filter=True, + ) + assert scores.date_ok is True + assert scores.date_inherited is True + assert scores.date_inherited_dataset == "dataset/transaction_date" + assert scores.all_ok is True + + +def test_no_actual_date_filter_still_passes_without_being_inherited(): + scores = check_filters(_viz(), _viz(), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True) + assert scores.date_ok is True + assert scores.date_inherited is False + + +def test_a_range_other_than_the_dashboards_fails(): + scores = check_filters( + _viz(), _viz({"f": _date(-5, 0)}), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True + ) + assert scores.date_ok is False + assert scores.date_inherited is False + + +@pytest.mark.parametrize( + ("expected_filter", "actual_filter", "date_ok"), + [ + # The question names a period: the dashboard's range is not good enough... + (_date(-1, -1, "YEAR"), _INHERITED, False), + # ...and the expected one still matches exactly, as before. + (_date(-1, -1, "YEAR"), _date(-1, -1, "YEAR"), True), + # Even an expected filter equal to the dashboard's is a strict match, not "inherited". + (_INHERITED, _INHERITED, True), + ], +) +def test_an_expected_date_filter_keeps_the_strict_comparison(expected_filter, actual_filter, date_ok): + expected, actual = _viz({"f": expected_filter}), _viz({"f": actual_filter}) + scores = check_filters(expected, actual, _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True) + assert scores.date_ok is date_ok + assert scores.date_inherited is False + assert scores == check_filters(expected, actual, _TODAY) + + +@pytest.mark.parametrize("user_context", [None, {}, {"view": {}}, {"view": {"dashboard": {"id": "d"}}}]) +def test_without_a_dashboard_date_filter_in_context_the_comparison_is_strict(user_context): + scores = check_filters( + _viz(), _viz({"f": _INHERITED}), _TODAY, user_context=user_context, accept_context_date_filter=True + ) + assert scores.date_ok is False + assert scores.date_inherited is False + + +# ── further edges of the rule ──────────────────────────────────────────────── + + +def test_any_date_dataset_is_accepted_and_recorded(): + scores = check_filters( + _viz(), + _viz({"f": _date(-11, 0, using="dataset/order_date")}), + _TODAY, + user_context=_USER_CONTEXT, + accept_context_date_filter=True, + ) + assert scores.date_ok is True + assert scores.date_inherited_dataset == "dataset/order_date" + + +@pytest.mark.parametrize(("actual_dataset", "date_ok"), [("dataset/order_date", True), ("dataset/ship_date", False)]) +def test_a_dashboard_filter_naming_a_dataset_accepts_only_that_one(actual_dataset, date_ok): + context = { + "view": { + "dashboard": { + "filters": [ + {"type": "date_filter", "using": "dataset/order_date", "granularity": "MONTH", "from": -11, "to": 0} + ] + } + } + } + scores = check_filters( + _viz(), + _viz({"f": _date(-11, 0, using=actual_dataset)}), + _TODAY, + user_context=context, + accept_context_date_filter=True, + ) + assert scores.date_ok is date_ok + assert scores.date_inherited is date_ok + + +def test_the_same_span_written_absolutely_is_accepted(): + # MONTH -11..0 on 2026-11-12 is 2025-12-01..2026-11-30. + absolute = {"type": "date_filter", "using": "dataset/transaction_date", "from": "2025-12-01", "to": "2026-11-30"} + scores = check_filters( + _viz(), _viz({"f": absolute}), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True + ) + assert scores.date_ok is True + assert scores.date_inherited is True + + +def test_two_actual_date_filters_fail_even_if_one_is_the_dashboards(): + actual = _viz({"a": _INHERITED, "b": _date(-1, -1, "YEAR", using="dataset/order_date")}) + scores = check_filters(_viz(), actual, _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True) + assert scores.date_ok is False + + +def test_any_of_several_dashboard_date_filters_matches(): + context = { + "view": { + "dashboard": { + "filters": [ + {"type": "attribute_filter", "using": "label/region"}, + {"type": "date_filter", "granularity": "YEAR", "from": 0, "to": 0}, + {"type": "date_filter", "granularity": "MONTH", "from": -11, "to": 0}, + ] + } + } + } + assert len(dashboard_date_filters(context)) == 2 + scores = check_filters( + _viz(), _viz({"f": _INHERITED}), _TODAY, user_context=context, accept_context_date_filter=True + ) + assert scores.date_ok is True + + +def test_ranking_and_attribute_filters_are_not_relaxed(): + attribute = {"type": "attribute_filter", "using": "label/region", "state": {"include": ["EMEA"]}} + scores = check_filters( + _viz(), + _viz({"f": _INHERITED, "r": attribute}), + _TODAY, + user_context=_USER_CONTEXT, + accept_context_date_filter=True, + ) + assert scores.date_ok is True + assert scores.attribute_ok is False + assert scores.all_ok is False + + +@pytest.mark.parametrize("malformed", ["text", {"view": "x"}, {"view": {"dashboard": {"filters": "x"}}}]) +def test_a_malformed_context_is_ignored_not_raised(malformed): + assert dashboard_date_filters(malformed) == [] + + +# ── the flag ──────────────────────────────────────────────────────────────── + + +@pytest.mark.parametrize("value", [None, "", "0", "false", "no", "off"]) +def test_flag_off_scores_strictly(monkeypatch, value): + if value is not None: + monkeypatch.setenv(_ENV, value) + expected, actual = _viz(), _viz({"f": _INHERITED}) + scores = check_filters(expected, actual, _TODAY, user_context=_USER_CONTEXT) + assert scores.date_ok is False + assert scores == check_filters(expected, actual, _TODAY) + + +@pytest.mark.parametrize("value", ["1", "true", "yes", "on"]) +def test_flag_on_from_the_environment(monkeypatch, value): + monkeypatch.setenv(_ENV, value) + scores = check_filters(_viz(), _viz({"f": _INHERITED}), _TODAY, user_context=_USER_CONTEXT) + assert scores.date_ok is True + assert scores.date_inherited is True + + +def test_the_keyword_overrides_the_environment(monkeypatch): + monkeypatch.setenv(_ENV, "1") + scores = check_filters( + _viz(), _viz({"f": _INHERITED}), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=False + ) + assert scores.date_ok is False + + +@pytest.mark.parametrize( + "actual_filter_by", + [{}, {"f": _INHERITED}, {"f": _date(-5, 0)}, {"a": _INHERITED, "b": _date(0, 0, "YEAR")}], +) +@pytest.mark.parametrize("expected_filter_by", [{}, {"f": _date(-1, -1, "YEAR")}]) +def test_flag_off_detail_is_byte_identical_with_and_without_user_context(expected_filter_by, actual_filter_by): + expected, actual = [_viz(expected_filter_by)], _viz(actual_filter_by) + before, _ = _evaluate_against_candidates(expected, actual, True, _TODAY) + after, _ = _evaluate_against_candidates(expected, actual, True, _TODAY, user_context=_USER_CONTEXT) + assert after == before + dump = json.dumps(evaluation_result_detail(before), sort_keys=True, default=sorted) + assert json.dumps(evaluation_result_detail(after), sort_keys=True, default=sorted) == dump + assert "filter_date_context" not in dump + + +# ── the real payload from the ticket, through both evaluation paths ────────── + + +def _ticket_item() -> DatasetItem: + return DatasetItem( + id="globalmart_visualization_charts_ctx_2", + dataset_name="globalmart_visualization_charts_ctx", + test_kind="visualization", + question="Show Gross Margin Percent as a headline.", + expected_output={"visualization": _headline()}, + user_context=_USER_CONTEXT, + ) + + +def _ticket_chat(metric: str = "metric/gross_margin_percent") -> ChatResult: + return ChatResult.model_validate( + { + "createdVisualizations": { + "objects": [_headline({"f_transaction_date": _INHERITED}, metric=metric)], + "reasoning": "", + }, + "toolCallEvents": [], + } + ) + + +def test_ticket_item_single_shot_fails_with_the_flag_off(): + result = get_evaluator("visualization").evaluate(_ticket_item(), _ticket_chat()) + assert result.passed is False + assert result.detail["filter_date_score"] is False + assert "filter_date_context" not in result.detail + + +def test_ticket_item_single_shot_passes_with_the_flag_on(monkeypatch): + monkeypatch.setenv(_ENV, "1") + result = get_evaluator("visualization").evaluate(_ticket_item(), _ticket_chat()) + assert result.passed is True + assert result.detail["filter_date_score"] is True + assert result.detail["filter_date_context"] == {"inherited": True, "dataset": "dataset/transaction_date"} + + +def _agentic_client(chat: ChatResult) -> MagicMock: + client = MagicMock() + client.create_conversation.return_value = "conv-1" + client.send_message.return_value = chat + return client + + +@pytest.mark.parametrize(("flag", "passes"), [(None, False), ("1", True)]) +def test_ticket_item_agentic_run(monkeypatch, flag, passes): + if flag is not None: + monkeypatch.setenv(_ENV, flag) + with patch.object(agentic_visualization, "ChatClient", return_value=_agentic_client(_ticket_chat())): + summary = run_agentic_visualization( + host="https://example.com", + token="tok", + workspace_id="ws", + question="Show Gross Margin Percent as a headline.", + expected_outputs=[_viz()], + k=1, + user_context=_USER_CONTEXT, + ) + ev = summary.best.eval_result + assert ev.strict_pass is passes + assert ev.filter_date_score is passes + assert ev.filter_date_inherited is passes + + +def test_strict_check_summary_shows_the_inherited_date(monkeypatch): + monkeypatch.setenv(_ENV, "1") + # A wrong metric keeps the item failing, so the summary is printed. + chat = _ticket_chat(metric="metric/revenue") + with ( + patch.object(agentic_visualization, "ChatClient", return_value=_agentic_client(chat)), + pytest.raises(VisualizationAssertionError) as exc_info, + ): + evaluate_agentic_visualization( + host="https://example.com", + token="tok", + workspace_id="ws", + question="Show Gross Margin Percent as a headline.", + expected_outputs=[_viz()], + k=1, + user_context=_USER_CONTEXT, + ) + message = str(exc_info.value) + assert " date : True (inherited from dashboard context)\n" in message + assert "Metrics Correct : False" in message + + +def test_strict_check_summary_is_unchanged_with_the_flag_off(): + with ( + patch.object(agentic_visualization, "ChatClient", return_value=_agentic_client(_ticket_chat())), + pytest.raises(VisualizationAssertionError) as exc_info, + ): + evaluate_agentic_visualization( + host="https://example.com", + token="tok", + workspace_id="ws", + question="Show Gross Margin Percent as a headline.", + expected_outputs=[_viz()], + k=1, + user_context=_USER_CONTEXT, + ) + message = str(exc_info.value) + assert " date : False\n" in message + assert "inherited" not in message + + +def test_failed_checks_leave_out_an_accepted_inherited_date(monkeypatch): + monkeypatch.setenv(_ENV, "1") + # An extra attribute filter fails the filters; the inherited date filter must not be blamed. + chat = ChatResult.model_validate( + { + "createdVisualizations": { + "objects": [ + _headline( + { + "f_transaction_date": _INHERITED, + "a": {"type": "attribute_filter", "using": "label/region", "state": {"include": ["x"]}}, + } + ) + ], + "reasoning": "", + }, + "toolCallEvents": [], + } + ) + with patch.object(agentic_visualization, "ChatClient", return_value=_agentic_client(chat)): + summary = run_agentic_visualization( + host="https://example.com", + token="tok", + workspace_id="ws", + question="Show Gross Margin Percent as a headline.", + expected_outputs=[_viz()], + k=1, + user_context=_USER_CONTEXT, + ) + run = summary.best + assert run.eval_result.filter_date_inherited is True + assert run.eval_result.filters_correct is False + labels = [c.get("label") for c in agentic_visualization._failed_checks(run) if c.get("name") == "filters_correct"] + assert "date" not in labels + assert "attribute" in labels diff --git a/packages/gooddata-eval/tests/test_eval_error_scores.py b/packages/gooddata-eval/tests/test_eval_error_scores.py new file mode 100644 index 000000000..f70fa5762 --- /dev/null +++ b/packages/gooddata-eval/tests/test_eval_error_scores.py @@ -0,0 +1,162 @@ +# (C) 2026 GoodData Corporation +"""Every agentic loop kind writes its Langfuse scores the same way (QA-30213, phase 1). + +A run labelled ERROR gets an extra ``eval_error`` score beside its usual ones; every other run, +a chat API error included, is scored exactly as before and gets none. The visualization kind +has its own tests in test_outcome.py. +""" + +from collections import defaultdict +from collections.abc import Callable, Iterator +from contextlib import ExitStack, contextmanager +from dataclasses import dataclass +from typing import Any +from unittest.mock import MagicMock, patch + +import pytest +from gooddata_eval.core.agentic import alert_skill, anomaly_detection, dashboard_skill, kda_skill, what_if +from gooddata_eval.core.agentic._outcome import RunLabels +from gooddata_eval.core.chat.sse_client import ChatError +from gooddata_eval.core.models import ChatResult +from gooddata_eval.core.outcome import Outcome + + +@dataclass(frozen=True) +class _Kind: + module: Any + evaluate: Callable[..., Any] + expected_output: dict + simulated_user: str | None + + +_KINDS = { + "alert_skill": _Kind( + alert_skill, + alert_skill.evaluate_agentic_alert_skill, + {"operator": "GREATER_THAN", "threshold": 100}, + "generate_simulated_alert_response", + ), + "kda_skill": _Kind( + kda_skill, + kda_skill.evaluate_agentic_kda_skill, + {"Measure": {"type": "metric", "id": "revenue"}}, + "generate_simulated_kda_response", + ), + "dashboard_skill": _Kind( + dashboard_skill, + dashboard_skill.evaluate_agentic_dashboard_skill, + { + "type": "dashboard", + "saved_dashboard_id": None, + "visualizations": [{"id": None, "title": "Revenue"}], + "date_range": None, + "min_new_visualizations": 1, + }, + None, + ), + "anomaly_detection": _Kind( + anomaly_detection, + anomaly_detection.evaluate_agentic_anomaly_detection, + {"metric": "metric/spend", "granularity": "MONTH"}, + "generate_simulated_anomaly_response", + ), + "what_if": _Kind( + what_if, + what_if.evaluate_agentic_what_if, + {"metric_id": "revenue"}, + "generate_simulated_what_if_response", + ), +} + + +class _ScoreCtx: + """A RunTraceContext stand-in that keeps each run's scores apart, by run index.""" + + def __init__(self) -> None: + self.run_metadata: dict[str, Any] = {} + self.scores: dict[int, list[tuple[str, Any, str | None]]] = defaultdict(list) + + def trace(self, _conversation_id: str) -> None: + return None + + def run_name(self, run_idx: int) -> str: + return f"run{run_idx}" + + @contextmanager + def observe(self, _trace: object, run_idx: int, **_kwargs: Any) -> Iterator[int]: + yield run_idx + + def score(self, trace_id: int, *, name: str, value: Any, data_type: str, comment: str | None = None) -> None: + self.scores[trace_id].append((name, value, comment)) + + def quality(self, trace_id: int, **_kwargs: Any) -> None: + self.scores[trace_id].append(("quality_score", None, None)) + + def names(self, run_idx: int) -> list[str]: + return [name for name, _, _ in self.scores[run_idx]] + + +def _scores(kind: _Kind, labels: RunLabels | None = None) -> _ScoreCtx: + """Run 1's chat call fails after its retries; run 2 answers without doing the task. + + ``labels`` replaces the labels the kind computes, to give run 1 a harness ERROR without + driving each kind's own simulated-user loop.""" + client = MagicMock() + client.create_conversation.side_effect = ["conv-1", "conv-2", "conv-3"] + client.send_message.side_effect = [ + ChatError("gen-ai fell over"), + ChatResult.model_validate({"textResponse": "I cannot help with that."}), + ] + captured: dict[str, Any] = {} + + def _capture(_submit: object, _identity: object, **kwargs: Any) -> None: + captured["write_scores"] = kwargs["write_scores"] + + with ExitStack() as stack: + stack.enter_context(patch.object(kind.module, "ChatClient", return_value=client)) + stack.enter_context(patch.object(kind.module, "submit_trace_scoring", _capture)) + if labels is not None: + stack.enter_context(patch.object(kind.module, "label_runs", return_value=labels)) + if kind.simulated_user is not None: + stack.enter_context(patch.object(kind.module, kind.simulated_user, return_value="please go on")) + try: + kind.evaluate( + "http://host/api/v1/actions/workspaces/ws1/ai", + "tok", + "ws1", + "Do the thing", + kind.expected_output, + k=2, + max_iterations=1, + langfuse=MagicMock(), + dataset_item_id="item-1", + ) + except AssertionError: + pass # the scored run fails; its scores are written before the raise + ctx = _ScoreCtx() + captured["write_scores"](ctx) + return ctx + + +_USUAL_SCORES = {"pass_at_k", "pass_power_k", "gate_passed"} + + +@pytest.mark.parametrize("name", sorted(_KINDS)) +def test_an_errored_run_gets_eval_error_beside_its_usual_scores(name) -> None: + labels = RunLabels((Outcome.error("simulated_user_failed"), Outcome.failed("agent_silent"))) + ctx = _scores(_KINDS[name], labels) + + assert ("eval_error", 1, "simulated_user_failed") in ctx.scores[0] + assert set(ctx.names(0)) >= _USUAL_SCORES + assert "eval_error" not in ctx.names(1) + assert set(ctx.names(1)) >= _USUAL_SCORES + + +@pytest.mark.parametrize("name", sorted(_KINDS)) +def test_a_chat_error_run_is_scored_as_before_with_no_marker(name) -> None: + ctx = _scores(_KINDS[name]) + + for run_idx in (0, 1): + names = ctx.names(run_idx) + assert "eval_error" not in names + assert set(names) >= _USUAL_SCORES diff --git a/packages/gooddata-eval/tests/test_outcome.py b/packages/gooddata-eval/tests/test_outcome.py new file mode 100644 index 000000000..4821dbd62 --- /dev/null +++ b/packages/gooddata-eval/tests/test_outcome.py @@ -0,0 +1,441 @@ +# (C) 2026 GoodData Corporation +"""PASS / FAIL / ERROR labels per run (QA-30213, phase 1): they explain, they never rescore. + +A run is ERROR only when the harness failed -- its simulated user or its judge. Every other +non-pass, a chat API error and the agent's own timeout included, is FAIL. The labels change no +gate, count or message head: an ERROR run still counts as a run that did not pass. +""" + +from collections import defaultdict +from collections.abc import Iterator +from contextlib import contextmanager +from typing import Any +from unittest.mock import MagicMock, patch + +import httpx +import pytest +from gooddata_eval.core.agentic import visualization as agentic_visualization +from gooddata_eval.core.agentic._gate import gate_failure_note +from gooddata_eval.core.agentic._outcome import chat_failure_exit, label_runs +from gooddata_eval.core.agentic.visualization import VisualizationAssertionError, evaluate_agentic_visualization +from gooddata_eval.core.chat.sse_client import ChatError, TransientChatError, TurnIncompleteError, TurnTimeoutError +from gooddata_eval.core.models import AgenticAssertionError, ChatResult, CreatedVisualization, LoopExit +from gooddata_eval.core.outcome import ( + JUDGE_FAILED, + Outcome, + OutcomeStatus, + classify_run, + item_outcome, + shadow_verdict, +) + +PASS = Outcome.passed() +FAIL = Outcome.failed("agent_silent") +SIM = Outcome.error("simulated_user_failed") +JUDGE = Outcome.error(JUDGE_FAILED) + + +# ── the rule: an allowlist of harness causes, everything else is FAIL ───────── + +# Every LoopExit, with the label of a run that ended on it without passing. Exhaustive on +# purpose: a new member must be classified here before the suite passes. +_EXPECTED = { + LoopExit.SUCCESS: OutcomeStatus.FAIL, + LoopExit.AGENT_SILENT: OutcomeStatus.FAIL, + LoopExit.BUDGET_EXHAUSTED: OutcomeStatus.FAIL, + LoopExit.STOPPED_EARLY: OutcomeStatus.FAIL, + LoopExit.SIMULATED_USER_FAILED: OutcomeStatus.ERROR, + LoopExit.CHAT_ERROR: OutcomeStatus.FAIL, + LoopExit.AGENT_TIMEOUT: OutcomeStatus.FAIL, + LoopExit.TURN_INCOMPLETE: OutcomeStatus.FAIL, + LoopExit.NOT_RUN: OutcomeStatus.FAIL, +} + + +def test_every_loop_exit_is_classified() -> None: + assert set(_EXPECTED) == set(LoopExit) + + +@pytest.mark.parametrize("loop_exit", list(LoopExit)) +def test_each_loop_exit_has_its_label_and_names_itself_as_the_reason(loop_exit: LoopExit) -> None: + outcome = classify_run(loop_exit, passed=False) + assert outcome.status is _EXPECTED[loop_exit] + assert outcome.reason == loop_exit.value + assert classify_run(loop_exit.value, passed=False) == outcome + + +@pytest.mark.parametrize("loop_exit", list(LoopExit)) +def test_a_run_that_passed_is_pass_whatever_its_exit(loop_exit: LoopExit) -> None: + assert classify_run(loop_exit, passed=True) == PASS + + +def test_a_judge_failure_is_an_error_whatever_the_exit() -> None: + assert classify_run(LoopExit.SUCCESS, passed=False, judge_error="empty body twice") == JUDGE + assert classify_run(None, passed=False, judge_error="unparseable JSON") == JUDGE + + +def test_an_unknown_cause_is_a_failure_never_an_error() -> None: + assert classify_run("some_new_exit", passed=False) == Outcome.failed("some_new_exit") + assert classify_run(None, passed=False) == Outcome.failed(None) + + +def test_an_outcome_record_is_json_ready() -> None: + assert SIM.as_record() == {"status": "error", "reason": "simulated_user_failed"} + assert Outcome.failed("chat_error").as_record() == {"status": "fail", "reason": "chat_error"} + + +@pytest.mark.parametrize( + ("exc", "loop_exit"), + [ + (TurnTimeoutError("agent exceeded the 60s turn budget"), LoopExit.AGENT_TIMEOUT), + (TurnIncompleteError("no final answer", reason="max iterations"), LoopExit.TURN_INCOMPLETE), + (TransientChatError("503 after retries"), LoopExit.CHAT_ERROR), + (ChatError("SSE stream error"), LoopExit.CHAT_ERROR), + (httpx.ReadError("connection reset"), LoopExit.CHAT_ERROR), + ], +) +def test_a_chat_failure_maps_to_its_own_exit_and_is_a_failure(exc: BaseException, loop_exit: LoopExit) -> None: + assert chat_failure_exit(exc) is loop_exit + assert classify_run(chat_failure_exit(exc), passed=False).status is OutcomeStatus.FAIL + + +# ── the item: ERROR only when no run was scored ────────────────────────────── + + +def test_an_item_is_an_error_only_when_every_run_errored() -> None: + assert item_outcome([SIM, SIM, SIM]) == SIM + assert item_outcome([SIM, PASS, PASS]) is None + assert item_outcome([SIM, FAIL]) is None + assert item_outcome([]) is None + + +def test_an_errored_item_names_its_most_frequent_reason() -> None: + assert item_outcome([SIM, JUDGE, JUDGE]) == JUDGE + + +# ── the shadow verdict: the phase-2 candidate rule, recorded and never used ── + + +@pytest.mark.parametrize( + ("outcomes", "gate", "verdict", "scored"), + [ + ([SIM, PASS, PASS], "power", "pass", 2), + ([SIM, SIM, PASS], "power", "error", 1), # fewer than ceil(3/2) scored runs + ([SIM, SIM, SIM], "power", "error", 0), + ([Outcome.failed("chat_error"), PASS, PASS], "power", "fail", 3), # a chat error is scored + ([SIM, PASS], "power", "pass", 1), # K=2 needs one scored run + ([SIM], "power", "error", 0), + ([SIM, FAIL, PASS], "any", "pass", 2), + ([PASS, PASS, PASS], "power", "pass", 3), + ], +) +def test_the_shadow_verdict(outcomes, gate, verdict, scored) -> None: + assert shadow_verdict(outcomes, gate) == {"verdict": verdict, "runs_scored": scored, "runs_total": len(outcomes)} + + +class _Run: + def __init__(self, ok: bool, exit_reason: LoopExit | None = None, judge_error: str | None = None) -> None: + self.ok = ok + self.exit_reason = exit_reason + self.judge_error = judge_error + + +def test_label_runs_reads_each_runs_exit_and_judge() -> None: + runs = [ + _Run(False, LoopExit.SIMULATED_USER_FAILED), + _Run(False, LoopExit.SUCCESS, judge_error="no verdict"), + _Run(True, LoopExit.SUCCESS), + _Run(False, LoopExit.CHAT_ERROR), + ] + labels = label_runs(runs, passed=lambda r: r.ok) + assert labels.outcomes == (SIM, JUDGE, PASS, Outcome.failed("chat_error")) + assert labels.errored_run(0) == SIM + assert labels.errored_run(2) is None + assert labels.errored_run(3) is None + assert labels.errored_run(9) is None + assert labels.item_outcome is None + assert labels.shadow("power") == {"verdict": "fail", "runs_scored": 2, "runs_total": 4} + + +# ── end to end through the visualization kind: labels, never a new verdict ─── + + +def _viz() -> dict: + return { + "id": "v1", + "type": "column_chart", + "query": { + "fields": {"m": {"using": "metric/revenue"}, "d": {"using": "label/date.quarter"}}, + "filter_by": {}, + }, + "metrics": ["m"], + "view_by": ["d"], + } + + +def _viz_turn() -> ChatResult: + return ChatResult.model_validate({"createdVisualizations": {"objects": [_viz()], "reasoning": ""}}) + + +def _question_turn() -> ChatResult: + return ChatResult.model_validate({"textResponse": "Which metric?"}) + + +def _evaluate(turns: list[Any], captured: dict[str, Any] | None = None) -> object: + client = MagicMock() + client.create_conversation.side_effect = [f"conv-{i}" for i in range(1, len(turns) + 1)] + client.send_message.side_effect = turns + + def _capture(_submit: object, _identity: object, **kwargs: Any) -> None: + if captured is not None: + captured["write_scores"] = kwargs["write_scores"] + + with ( + patch.object(agentic_visualization, "ChatClient", return_value=client), + patch.object(agentic_visualization, "generate_simulated_response", side_effect=RuntimeError("openai 401")), + patch.object(agentic_visualization, "submit_trace_scoring", _capture), + ): + return evaluate_agentic_visualization( + host="https://example.com", + token="tok", + workspace_id="ws", + question="Show revenue by quarter", + expected_outputs=[CreatedVisualization.model_validate(_viz())], + k=len(turns), + gate="power", + langfuse=MagicMock() if captured is not None else None, + dataset_item_id="item-1" if captured is not None else None, + ) + + +def _silent_turn() -> ChatResult: + return ChatResult.model_validate({}) + + +def test_a_simulated_user_failure_keeps_the_verdict_and_is_labelled() -> None: + with pytest.raises(VisualizationAssertionError) as exc_info: + _evaluate([_question_turn(), _viz_turn(), _viz_turn()]) + + exc = exc_info.value + message = str(exc) + lines = message.splitlines() + assert lines[1] == "Agentic Visualization Assertion Failed! (Critical Mode)" + # The gate counted the errored run as a run that did not pass: pass^3, 2/3, as before. + assert lines[2].startswith(gate_failure_note("power", 2, 3)) + assert "run 1/3 ERROR (simulated_user_failed)" in lines[2] + assert (exc.runs_passed, exc.runs_effective) == (2, 3) + (record,) = exc.failed_runs + assert record["exit_reason"] == "simulated_user_failed" + assert record["outcome"] == {"status": "error", "reason": "simulated_user_failed"} + # Some runs were scored, so the item is not a harness error. + assert not hasattr(exc, "outcome") + assert exc.shadow == {"verdict": "pass", "runs_scored": 2, "runs_total": 3} + + +def test_an_item_whose_every_run_errored_is_labelled_and_still_fails() -> None: + with pytest.raises(VisualizationAssertionError) as exc_info: + _evaluate([_question_turn(), _question_turn(), _question_turn()]) + + exc = exc_info.value + assert isinstance(exc, AgenticAssertionError) + assert exc.outcome == {"status": "error", "reason": "simulated_user_failed"} + assert exc.shadow == {"verdict": "error", "runs_scored": 0, "runs_total": 3} + gate_line = str(exc).splitlines()[2] + assert gate_line.startswith( + f"Harness error, no run was scored (simulated_user_failed). {gate_failure_note('power', 0, 3)}" + ) + assert exc.runs_passed == 0 + assert len(exc.failed_runs) == 3 + + +@pytest.mark.parametrize( + ("exc", "exit_value"), + [ + (TurnTimeoutError("agent exceeded the 60s turn budget"), "agent_timeout"), + (TurnIncompleteError("no final answer", reason="max iterations"), "turn_incomplete"), + (ChatError("SSE stream error"), "chat_error"), + ], +) +def test_a_chat_failure_is_a_plain_failure(exc, exit_value) -> None: + with pytest.raises(VisualizationAssertionError) as exc_info: + _evaluate([exc, _viz_turn(), _viz_turn()]) + + raised = exc_info.value + assert (raised.runs_passed, raised.runs_effective) == (2, 3) + (record,) = raised.failed_runs + assert record["exit_reason"] == exit_value + assert record["outcome"] == {"status": "fail", "reason": exit_value} + assert f"run 1/3 FAIL (loop exit: {exit_value})" in str(raised) + assert "ERROR" not in str(raised) + assert not hasattr(raised, "outcome") + assert raised.shadow["verdict"] == "fail" + + +def test_only_timeouts_is_a_failure_not_a_harness_error() -> None: + timeout = TurnTimeoutError("agent exceeded the 60s turn budget") + with pytest.raises(VisualizationAssertionError) as exc_info: + _evaluate([timeout, timeout, timeout]) + + assert not hasattr(exc_info.value, "outcome") + assert str(exc_info.value).splitlines()[2].startswith(gate_failure_note("power", 0, 3)) + + +# ── Langfuse: an ERROR run gets an extra marker, every usual score stays ────── + + +class _ScoreCtx: + """A RunTraceContext stand-in that keeps each run's scores apart, by run index.""" + + def __init__(self) -> None: + self.run_metadata: dict[str, Any] = {} + self.scores: dict[int, list[tuple[str, Any, str | None]]] = defaultdict(list) + + def trace(self, _conversation_id: str) -> None: + return None + + @contextmanager + def observe(self, _trace: object, run_idx: int, **_kwargs: Any) -> Iterator[int]: + yield run_idx + + def score(self, trace_id: int, *, name: str, value: Any, data_type: str, comment: str | None = None) -> None: + self.scores[trace_id].append((name, value, comment)) + + def quality(self, trace_id: int, **_kwargs: Any) -> None: + self.scores[trace_id].append(("quality_score", None, None)) + + def names(self, run_idx: int) -> list[str]: + return [name for name, _, _ in self.scores[run_idx]] + + +def _scores(turns: list[Any]) -> _ScoreCtx: + captured: dict[str, Any] = {} + try: + _evaluate(turns, captured) + except AssertionError: + pass + ctx = _ScoreCtx() + captured["write_scores"](ctx) + return ctx + + +_USUAL_SCORES = {"gate_passed", "pass_at_k", "pass_power_k", "quality_score"} + + +def test_an_errored_run_gets_eval_error_beside_its_usual_scores() -> None: + ctx = _scores([_question_turn(), _viz_turn(), _viz_turn()]) + + assert ("eval_error", 1, "simulated_user_failed") in ctx.scores[0] + assert set(ctx.names(0)) >= _USUAL_SCORES + # The verdict scores are the ones the item was judged by: unchanged by the label. + assert dict((n, v) for n, v, _ in ctx.scores[0])["gate_passed"] is False + for run_idx in (1, 2): + assert "eval_error" not in ctx.names(run_idx) + assert set(ctx.names(run_idx)) >= _USUAL_SCORES + + +def test_a_chat_error_run_gets_no_marker() -> None: + ctx = _scores([ChatError("SSE stream error"), _viz_turn(), _viz_turn()]) + + for run_idx in range(3): + assert "eval_error" not in ctx.names(run_idx) + assert set(ctx.names(run_idx)) >= _USUAL_SCORES + + +# ── the message is the master text plus the always-on failing runs ─────────── + +# The message of [simulated user failed, agent silent, pass] under pass^3, as the package +# produced it before labels existed. Only the "Failing runs" part of the gate line is new. +_MASTER_MESSAGE = ( + "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n" + "Agentic Visualization Assertion Failed! (Critical Mode)\n" + "Gate pass^3 failed: 1/3 runs passed — unstable, not a clean failure.\n" + "------------------------------------------\n" + "Question:\n" + "Show revenue by quarter\n" + "------------------------------------------\n" + "Expected Output:\n" + "{\n" + ' "id": "v1",\n' + ' "type": "column_chart",\n' + ' "query": {\n' + ' "fields": {\n' + ' "m": {\n' + ' "using": "metric/revenue"\n' + " },\n" + ' "d": {\n' + ' "using": "label/date.quarter"\n' + " }\n" + " },\n" + ' "filter_by": {},\n' + ' "sort_by": []\n' + " },\n" + ' "metrics": [\n' + ' "m"\n' + " ],\n" + ' "view_by": [\n' + ' "d"\n' + " ],\n" + ' "segment_by": [],\n' + ' "rows": [],\n' + ' "columns": [],\n' + ' "size_by": []\n' + "}\n" + "------------------------------------------\n" + "Actual Output:\n" + "{\n" + ' "id": "v1",\n' + ' "type": "column_chart",\n' + ' "query": {\n' + ' "fields": {\n' + ' "m": {\n' + ' "using": "metric/revenue"\n' + " },\n" + ' "d": {\n' + ' "using": "label/date.quarter"\n' + " }\n" + " },\n" + ' "filter_by": {},\n' + ' "sort_by": []\n' + " },\n" + ' "metrics": [\n' + ' "m"\n' + " ],\n" + ' "view_by": [\n' + ' "d"\n' + " ],\n" + ' "segment_by": [],\n' + ' "rows": [],\n' + ' "columns": [],\n' + ' "size_by": []\n' + "}\n" + "------------------------------------------\n" + "Strict Check Summary:\n" + " Visualization Created : True\n" + " Cross-Ref Valid : True\n" + " Metrics Correct : True\n" + " expected : ['metric/revenue']\n" + " actual : ['metric/revenue']\n" + " Dimensions Correct : True\n" + " expected : ['label/date.quarter']\n" + " actual : ['label/date.quarter']\n" + " Filters Correct : True\n" + " date : True\n" + " ranking : True\n" + " attribute : True\n" + " Viz Type Hard : True\n" + " Executed : False\n" + " Stated Value Matches : None (reported only)\n" + "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n" +) + + +def test_the_agentic_message_is_the_master_text_plus_the_failing_runs() -> None: + with pytest.raises(VisualizationAssertionError) as exc_info: + _evaluate([_question_turn(), _silent_turn(), _viz_turn()]) + + message = str(exc_info.value) + gate_line = message.splitlines()[2] + head, failing = gate_line.split(" Failing runs: ", 1) + assert failing.startswith("run 1/3 ERROR (simulated_user_failed); ") + assert "; run 2/3 FAIL (loop exit: agent_silent): " in failing + assert message.replace(gate_line, head, 1) == _MASTER_MESSAGE From 3aabc8f8f6bc8f7e70f4606f2634cf6f018f81d0 Mon Sep 17 00:00:00 2001 From: "my.nguyen" <my.nguyen@gooddata.com> Date: Tue, 17 Nov 2026 13:56:04 +0700 Subject: [PATCH 2/5] fix(gooddata-eval): mark judge-failed runs, keep request failures FAIL Address review findings on the phase-1 labels: - A run the judge could not grade (general question, guardrail, document skill) now gets its run root with the eval_error marker instead of no score at all; other unscored runs keep only their root, as before. - The document skill labels from the judge's own error. Its judge_error property also reports a failed takeaway request, which is on the product path and stays FAIL. - An inherited date filter is accepted only when the chart's filter names a dataset. - Annotate the new test functions. jira: QA-30213 risk: low Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- .../core/agentic/_failed_runs.py | 7 ++- .../gooddata_eval/core/agentic/_outcome.py | 40 ++++++++++--- .../core/agentic/document_skill.py | 23 ++++++-- .../core/agentic/general_question.py | 8 ++- .../gooddata_eval/core/agentic/guardrail.py | 8 ++- .../src/gooddata_eval/core/scoring.py | 7 ++- .../tests/test_agentic_document_skill.py | 17 ++++-- .../tests/test_agentic_failure_message.py | 2 +- packages/gooddata-eval/tests/test_cli.py | 8 ++- .../tests/test_context_date_filter.py | 57 ++++++++++++------- .../tests/test_eval_error_scores.py | 4 +- packages/gooddata-eval/tests/test_outcome.py | 51 ++++++++++++++++- 12 files changed, 171 insertions(+), 61 deletions(-) diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failed_runs.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failed_runs.py index 300121e08..fd056088d 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failed_runs.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_failed_runs.py @@ -26,7 +26,7 @@ from typing import Any from gooddata_eval.core.agentic._failure_message import cap_failed_checks -from gooddata_eval.core.agentic._outcome import run_exit_reason, run_outcome +from gooddata_eval.core.agentic._outcome import JudgeErrorOf, run_exit_reason, run_outcome from gooddata_eval.core.models import build_tool_calls @@ -36,6 +36,7 @@ def build_failed_runs( passed: Callable[[Any], bool], detail: Callable[[Any], dict], failed_checks: Callable[[Any], list[dict]] | None = None, + judge_error: JudgeErrorOf | None = None, ) -> list[dict]: """One record per run that did not pass, in the order the runs happened. @@ -54,7 +55,7 @@ def build_failed_runs( ``exit_reason`` is the run's ``LoopExit`` value, or null for a kind whose loop records none, and ``outcome`` its ``core.outcome.Outcome`` record: ERROR when the harness failed (its simulated user or its judge), FAIL otherwise. The label decides nothing; the run - counts as not passed either way. ``failed_checks`` is written only when the caller supplies it: a missing + counts as not passed either way; ``judge_error`` is passed on to ``run_outcome``. ``failed_checks`` is written only when the caller supplies it: a missing key means the kind does not name its failing checks, while an empty list would claim the run failed none. Its expected/actual values are capped in size; the run's ``detail`` keeps them whole. @@ -78,7 +79,7 @@ def build_failed_runs( "tool_names": [getattr(e, "function_name", None) for e in tool_calls], "tool_calls": build_tool_calls(tool_calls), "exit_reason": run_exit_reason(run), - "outcome": run_outcome(run, passed=False).as_record(), + "outcome": run_outcome(run, passed=False, judge_error=judge_error).as_record(), } if failed_checks is not None: record["failed_checks"] = cap_failed_checks(failed_checks(run)) diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_outcome.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_outcome.py index f45c33290..0a663675a 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/_outcome.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/_outcome.py @@ -49,6 +49,7 @@ "gate_head", "label_runs", "log_eval_error_score", + "mark_unscored_run", "raise_agentic_failure", "run_exit_reason", "run_outcome", @@ -83,12 +84,20 @@ def run_exit_reason(run: Any) -> str | None: return value if isinstance(value, str) else None -def run_outcome(run: Any, *, passed: bool) -> Outcome: - """One run's label: ``classify_run`` over the run's exit and its judge error, if any.""" - judge_error = getattr(run, "judge_error", None) - return classify_run( - run_exit_reason(run), passed=passed, judge_error=judge_error if isinstance(judge_error, str) else None - ) +JudgeErrorOf = Callable[[Any], str | None] +"""How a kind reads a run's judge failure. The default reads ``run.judge_error``; a kind whose +``judge_error`` also reports a product-path failure passes the judge's own error instead.""" + + +def _default_judge_error(run: Any) -> str | None: + value = getattr(run, "judge_error", None) + return value if isinstance(value, str) else None + + +def run_outcome(run: Any, *, passed: bool, judge_error: JudgeErrorOf | None = None) -> Outcome: + """One run's label: ``classify_run`` over the run's exit and its judge failure, if any.""" + error = (judge_error or _default_judge_error)(run) + return classify_run(run_exit_reason(run), passed=passed, judge_error=error if isinstance(error, str) else None) @dataclass(frozen=True) @@ -123,9 +132,11 @@ def shadow(self, gate: str | None) -> dict[str, Any]: return shadow_verdict(self.outcomes, gate) -def label_runs(run_results: Sequence[Any], *, passed: Callable[[Any], bool]) -> RunLabels: +def label_runs( + run_results: Sequence[Any], *, passed: Callable[[Any], bool], judge_error: JudgeErrorOf | None = None +) -> RunLabels: """The ``RunLabels`` of an item's runs; ``passed`` is the kind's own per-run verdict.""" - return RunLabels(tuple(run_outcome(r, passed=passed(r)) for r in run_results)) + return RunLabels(tuple(run_outcome(r, passed=passed(r), judge_error=judge_error) for r in run_results)) def attach_labels(exc: BaseException, labels: RunLabels, gate: str | None) -> None: @@ -162,6 +173,19 @@ def log_eval_error_score(ctx: Any, trace_id: Any, outcome: Outcome) -> None: ctx.score(trace_id, name=EVAL_ERROR_SCORE, value=1, data_type="NUMERIC", comment=outcome.reason) +def mark_unscored_run(ctx: Any, run_idx: int, conversation_id: str, error: Outcome | None) -> None: + """Langfuse for a run that gets no verdict scores (its judge or its request failed). + + A harness ERROR (``error``, from ``RunLabels.errored_run``) gets its run root with the + ``eval_error`` marker; any other unscored run only keeps its root, as before. + """ + if error is None: + ctx.export_unscored(run_idx, conversation_id) + return + with ctx.observe(ctx.trace(conversation_id), run_idx, conversation_id=conversation_id) as tid: + log_eval_error_score(ctx, tid, error) + + def agentic_detail( tool_call_events: list[ToolCallEvent], reasoning_step_events: list[ReasoningStepEvent] | None, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py index 96f3c3e9a..fd8d46377 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py @@ -24,7 +24,7 @@ log_gate_scores, stamp_gate_metadata, ) -from gooddata_eval.core.agentic._outcome import attach_labels, gate_head, label_runs +from gooddata_eval.core.agentic._outcome import attach_labels, gate_head, label_runs, mark_unscored_run from gooddata_eval.core.agentic._trace_linker import ( RunIdentity, RunTraceContext, @@ -1716,6 +1716,12 @@ class DocumentSkillAssertionError(AgenticAssertionError): """Raised when a document-skill evaluation fails.""" +def _judge_error_of(run: DocumentRunResult) -> str | None: + """The judge's own failure, for the harness label. ``run.judge_error`` also reports a failed + takeaway request (``fill_error``), which is on the product path and stays a FAIL.""" + return run.evaluation.judge_error + + def evaluate_agentic_document_skill( host: str, token: str, @@ -1795,6 +1801,9 @@ def evaluate_agentic_document_skill( document_format=document_format, ) + # Before the Langfuse scores, so a run the harness failed gets its marker there too. + run_labels = label_runs(summary.run_results, passed=lambda r: r.evaluation.strict_pass, judge_error=_judge_error_of) + if langfuse is not None and dataset_item_id: # Pinned on the calling thread: a deferred poll must not widen its query window. window_end = utc_now() @@ -1806,7 +1815,7 @@ def _write_scores(ctx: RunTraceContext) -> None: if run.ungraded: # No verdict decided this run: its other scores would publish a pass the gate # never counted. - ctx.export_unscored(run_idx, run.conversation_id) + mark_unscored_run(ctx, run_idx, run.conversation_id, run_labels.errored_run(run_idx)) continue pt = ctx.trace(run.conversation_id) strict_checks = run.evaluation.strict_checks @@ -1856,9 +1865,6 @@ def _write_scores(ctx: RunTraceContext) -> None: unscored = summary.unscored_errors judge_errors = summary.judge_errors request_errors = summary.request_errors - # A run the judge or scorer could not grade is labelled ERROR (judge_failed); a run whose - # request failed is on the product path and stays FAIL. The gate is unchanged. - run_labels = label_runs(summary.run_results, passed=lambda r: r.evaluation.strict_pass) if not summary.scored_run_results: causes = [ *([f"judge returned no readable verdict for {len(judge_errors)} run(s)"] if judge_errors else []), @@ -1900,7 +1906,12 @@ def _run_detail(run: DocumentRunResult) -> dict[str, Any]: } # Same predicate runs_passed is taken over, so an item's failed_runs and its counts # cannot disagree about which runs failed. - failed_runs = build_failed_runs(summary.run_results, passed=lambda r: r.evaluation.strict_pass, detail=_run_detail) + failed_runs = build_failed_runs( + summary.run_results, + passed=lambda r: r.evaluation.strict_pass, + detail=_run_detail, + judge_error=_judge_error_of, + ) if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): gate_note = gate_head(gate, run_labels, runs_passed, failed_runs, len(unscored)) diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/general_question.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/general_question.py index c1e3c8e87..af50b3cf6 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/general_question.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/general_question.py @@ -20,6 +20,7 @@ attach_labels, gate_head, label_runs, + mark_unscored_run, raise_agentic_failure, ) from gooddata_eval.core.agentic._trace_linker import ( @@ -345,6 +346,9 @@ def evaluate_agentic_general_question( document_format=document_format, ) + # Before the Langfuse scores, so a run the harness failed gets its marker there too. + run_labels = label_runs(summary.run_results, passed=lambda r: r.judge_error is None and r.passed) + if langfuse is not None and dataset_item_id: # Pinned on the calling thread: a deferred poll must not widen its query window. window_end = utc_now() @@ -356,7 +360,7 @@ def _write_scores(ctx: RunTraceContext) -> None: if run.judge_error is not None: # No verdict for this run: float(run.passed) would write a 0 the judge # never returned. - ctx.export_unscored(run_idx, run.conversation_id) + mark_unscored_run(ctx, run_idx, run.conversation_id, run_labels.errored_run(run_idx)) continue pt = ctx.trace(run.conversation_id) with ctx.observe( @@ -423,8 +427,6 @@ def _write_scores(ctx: RunTraceContext) -> None: passed=lambda r: r.judge_error is None and r.passed, detail=_run_detail, ) - # A run the judge could not grade is labelled ERROR (judge_failed); the gate is unchanged. - run_labels = label_runs(summary.run_results, passed=lambda r: r.judge_error is None and r.passed) if not summary.scored_run_results: # Not one run produced a readable verdict, so this item has no result -- an error, diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/guardrail.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/guardrail.py index 64d835ab8..37e46fd37 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/guardrail.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/guardrail.py @@ -20,6 +20,7 @@ attach_labels, gate_head, label_runs, + mark_unscored_run, raise_agentic_failure, ) from gooddata_eval.core.agentic._trace_linker import ( @@ -305,6 +306,9 @@ def evaluate_agentic_guardrail( user_context=user_context, ) + # Before the Langfuse scores, so a run the harness failed gets its marker there too. + run_labels = label_runs(summary.run_results, passed=lambda r: r.judge_error is None and r.passed) + if langfuse is not None and dataset_item_id: # Pinned on the calling thread: a deferred poll must not widen its query window. window_end = utc_now() @@ -316,7 +320,7 @@ def _write_scores(ctx: RunTraceContext) -> None: if run.judge_error is not None: # No verdict, so nothing truthful to publish: float(run.passed) would # write a 0 the judge never returned. - ctx.export_unscored(run_idx, run.conversation_id) + mark_unscored_run(ctx, run_idx, run.conversation_id, run_labels.errored_run(run_idx)) continue pt = ctx.trace(run.conversation_id) with ctx.observe( @@ -381,8 +385,6 @@ def _write_scores(ctx: RunTraceContext) -> None: passed=lambda r: r.judge_error is None and r.passed, detail=_run_detail, ) - # A run the judge could not grade is labelled ERROR (judge_failed); the gate is unchanged. - run_labels = label_runs(summary.run_results, passed=lambda r: r.judge_error is None and r.passed) if not summary.scored_run_results: # No readable verdict for any run: an error, not K failures. Raised after the diff --git a/packages/gooddata-eval/src/gooddata_eval/core/scoring.py b/packages/gooddata-eval/src/gooddata_eval/core/scoring.py index b6fbc99cd..912fb837f 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/scoring.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/scoring.py @@ -551,14 +551,17 @@ def _inherited_date_filter_dataset( actual_filters = [f for f in actual.query.filter_by.values() if f.get("type") == "date_filter"] if len(actual_filters) != 1: return None - dataset = actual_filters[0].get("using", "") + dataset = actual_filters[0].get("using") + if not isinstance(dataset, str) or not dataset: + # A date filter that names no dataset says nothing about where it came from. + return None fields = actual.query.fields for dashboard_filter in dashboard_filters: candidate = _normalize_date_filter( {**dashboard_filter, "using": dashboard_filter.get("using") or dataset}, fields, today ) if json.dumps(candidate, sort_keys=True) in act_date: - return dataset if isinstance(dataset, str) else "" + return dataset return None diff --git a/packages/gooddata-eval/tests/test_agentic_document_skill.py b/packages/gooddata-eval/tests/test_agentic_document_skill.py index e1a0c0f2d..78d79410a 100644 --- a/packages/gooddata-eval/tests/test_agentic_document_skill.py +++ b/packages/gooddata-eval/tests/test_agentic_document_skill.py @@ -1733,6 +1733,7 @@ def __init__(self) -> None: self.quality_checks: dict[str, bool] = {} self.observed: list[str] = [] self.unscored: list[str] = [] + self.comments: dict[str, str] = {} def export_unscored(self, _run_idx: int, conversation_id: str) -> None: self.unscored.append(conversation_id) @@ -1741,13 +1742,17 @@ def trace(self, _conversation_id: str) -> None: return None @contextmanager - def observe(self, _trace: None, _run_idx: int, *, conversation_id: str, output: dict) -> Iterator[str]: + def observe( + self, _trace: None, _run_idx: int, *, conversation_id: str, output: dict | None = None + ) -> Iterator[str]: self.observed.append(conversation_id) yield "trace-id" - def score(self, _tid: str, *, name: str, value: float, data_type: str) -> None: + def score(self, _tid: str, *, name: str, value: float, data_type: str, comment: str | None = None) -> None: self.scores[name] = value self.score_types[name] = data_type + if comment is not None: + self.comments[name] = comment def quality( self, _tid: str, *, strict_checks: dict[str, bool], latency_sec: float | None, cost_usd: float | None @@ -1991,10 +1996,12 @@ def test_an_ungraded_run_keeps_pass_at_k_but_not_pass_power_k(monkeypatch: pytes assert summary.best.judge_error is None -def test_an_ungraded_run_writes_no_scores(monkeypatch: pytest.MonkeyPatch) -> None: +def test_a_run_the_judge_could_not_grade_gets_only_the_harness_marker(monkeypatch: pytest.MonkeyPatch) -> None: ctx = _scored(monkeypatch, {"narrative": _NARRATIVE}, _narrative_turns(), k=2, judge=_SequenceJudge()) - assert ctx.observed == ["conv-2"] - assert ctx.unscored == ["conv-1"] + # conv-1 has no verdict: its root carries eval_error alone; conv-2 is scored as before. + assert ctx.observed == ["conv-1", "conv-2"] + assert ctx.unscored == [] + assert (ctx.scores["eval_error"], ctx.comments["eval_error"]) == (1, "judge_failed") assert ctx.scores["document_narrative_judged"] == 1.0 assert ctx.scores["document_summaries_present"] == 1.0 diff --git a/packages/gooddata-eval/tests/test_agentic_failure_message.py b/packages/gooddata-eval/tests/test_agentic_failure_message.py index 0d60acc20..22907fd7c 100644 --- a/packages/gooddata-eval/tests/test_agentic_failure_message.py +++ b/packages/gooddata-eval/tests/test_agentic_failure_message.py @@ -146,7 +146,7 @@ def test_a_string_exit_reason_gets_the_same_outcome_as_its_enum() -> None: @pytest.mark.parametrize( "exit_reason", [LoopExit.CHAT_ERROR, "chat_error", "budget_exhausted"], ids=["member", "value", "agent_exit"] ) -def test_a_chat_error_or_agent_exit_is_recorded_and_rendered_as_a_failure(exit_reason) -> None: +def test_a_chat_error_or_agent_exit_is_recorded_and_rendered_as_a_failure(exit_reason: LoopExit | str) -> None: records = build_failed_runs( [_StrRun(exit_reason)], passed=lambda r: False, detail=lambda r: {}, failed_checks=lambda r: [] ) diff --git a/packages/gooddata-eval/tests/test_cli.py b/packages/gooddata-eval/tests/test_cli.py index 3987f98e3..62af25bd4 100644 --- a/packages/gooddata-eval/tests/test_cli.py +++ b/packages/gooddata-eval/tests/test_cli.py @@ -1206,7 +1206,9 @@ def test_routing_backend_closes_every_client() -> None: client.close.assert_called_once_with() -def test_accept_context_date_filter_flag_is_off_by_default_and_publishes_the_env_var(monkeypatch): +def test_accept_context_date_filter_flag_is_off_by_default_and_publishes_the_env_var( + monkeypatch: pytest.MonkeyPatch, +) -> None: base = ["run", "--host", "h", "--workspace", "w", "--dataset", "d"] assert cli_main.parse_args(base).accept_context_date_filter is False assert cli_main.parse_args([*base, "--accept-context-date-filter"]).accept_context_date_filter is True @@ -1219,13 +1221,13 @@ def test_accept_context_date_filter_flag_is_off_by_default_and_publishes_the_env assert accept_context_date_filter() is True -def test_accept_context_date_filter_flag_never_clears_an_exported_env_var(monkeypatch): +def test_accept_context_date_filter_flag_never_clears_an_exported_env_var(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setenv(ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, "1") cli_main._apply_accept_context_date_filter_flag(False) assert accept_context_date_filter() is True -def test_accept_context_date_filter_flag_reaches_check_filters(monkeypatch): +def test_accept_context_date_filter_flag_reaches_check_filters(monkeypatch: pytest.MonkeyPatch) -> None: def viz(filter_by: dict) -> CreatedVisualization: return CreatedVisualization.model_validate( { diff --git a/packages/gooddata-eval/tests/test_context_date_filter.py b/packages/gooddata-eval/tests/test_context_date_filter.py index 7f9df3753..f66f1004f 100644 --- a/packages/gooddata-eval/tests/test_context_date_filter.py +++ b/packages/gooddata-eval/tests/test_context_date_filter.py @@ -63,7 +63,7 @@ def _date(frm: int, to: int, granularity: str = "MONTH", using: str = "dataset/t # ── the five acceptance-criteria cases (flag on) ───────────────────────────── -def test_inherited_dashboard_range_passes_and_is_marked_inherited(): +def test_inherited_dashboard_range_passes_and_is_marked_inherited() -> None: scores = check_filters( _viz(), _viz({"f_transaction_date": _INHERITED}), @@ -77,13 +77,13 @@ def test_inherited_dashboard_range_passes_and_is_marked_inherited(): assert scores.all_ok is True -def test_no_actual_date_filter_still_passes_without_being_inherited(): +def test_no_actual_date_filter_still_passes_without_being_inherited() -> None: scores = check_filters(_viz(), _viz(), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True) assert scores.date_ok is True assert scores.date_inherited is False -def test_a_range_other_than_the_dashboards_fails(): +def test_a_range_other_than_the_dashboards_fails() -> None: scores = check_filters( _viz(), _viz({"f": _date(-5, 0)}), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True ) @@ -102,7 +102,9 @@ def test_a_range_other_than_the_dashboards_fails(): (_INHERITED, _INHERITED, True), ], ) -def test_an_expected_date_filter_keeps_the_strict_comparison(expected_filter, actual_filter, date_ok): +def test_an_expected_date_filter_keeps_the_strict_comparison( + expected_filter: dict, actual_filter: dict, date_ok: bool +) -> None: expected, actual = _viz({"f": expected_filter}), _viz({"f": actual_filter}) scores = check_filters(expected, actual, _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True) assert scores.date_ok is date_ok @@ -111,7 +113,7 @@ def test_an_expected_date_filter_keeps_the_strict_comparison(expected_filter, ac @pytest.mark.parametrize("user_context", [None, {}, {"view": {}}, {"view": {"dashboard": {"id": "d"}}}]) -def test_without_a_dashboard_date_filter_in_context_the_comparison_is_strict(user_context): +def test_without_a_dashboard_date_filter_in_context_the_comparison_is_strict(user_context: dict | None) -> None: scores = check_filters( _viz(), _viz({"f": _INHERITED}), _TODAY, user_context=user_context, accept_context_date_filter=True ) @@ -122,7 +124,7 @@ def test_without_a_dashboard_date_filter_in_context_the_comparison_is_strict(use # ── further edges of the rule ──────────────────────────────────────────────── -def test_any_date_dataset_is_accepted_and_recorded(): +def test_any_date_dataset_is_accepted_and_recorded() -> None: scores = check_filters( _viz(), _viz({"f": _date(-11, 0, using="dataset/order_date")}), @@ -134,8 +136,17 @@ def test_any_date_dataset_is_accepted_and_recorded(): assert scores.date_inherited_dataset == "dataset/order_date" +def test_a_date_filter_that_names_no_dataset_is_never_inherited() -> None: + nameless = {k: v for k, v in _date(-11, 0).items() if k != "using"} + scores = check_filters( + _viz(), _viz({"f": nameless}), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True + ) + assert scores.date_ok is False + assert scores.date_inherited is False + + @pytest.mark.parametrize(("actual_dataset", "date_ok"), [("dataset/order_date", True), ("dataset/ship_date", False)]) -def test_a_dashboard_filter_naming_a_dataset_accepts_only_that_one(actual_dataset, date_ok): +def test_a_dashboard_filter_naming_a_dataset_accepts_only_that_one(actual_dataset: str, date_ok: bool) -> None: context = { "view": { "dashboard": { @@ -156,7 +167,7 @@ def test_a_dashboard_filter_naming_a_dataset_accepts_only_that_one(actual_datase assert scores.date_inherited is date_ok -def test_the_same_span_written_absolutely_is_accepted(): +def test_the_same_span_written_absolutely_is_accepted() -> None: # MONTH -11..0 on 2026-11-12 is 2025-12-01..2026-11-30. absolute = {"type": "date_filter", "using": "dataset/transaction_date", "from": "2025-12-01", "to": "2026-11-30"} scores = check_filters( @@ -166,13 +177,13 @@ def test_the_same_span_written_absolutely_is_accepted(): assert scores.date_inherited is True -def test_two_actual_date_filters_fail_even_if_one_is_the_dashboards(): +def test_two_actual_date_filters_fail_even_if_one_is_the_dashboards() -> None: actual = _viz({"a": _INHERITED, "b": _date(-1, -1, "YEAR", using="dataset/order_date")}) scores = check_filters(_viz(), actual, _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True) assert scores.date_ok is False -def test_any_of_several_dashboard_date_filters_matches(): +def test_any_of_several_dashboard_date_filters_matches() -> None: context = { "view": { "dashboard": { @@ -191,7 +202,7 @@ def test_any_of_several_dashboard_date_filters_matches(): assert scores.date_ok is True -def test_ranking_and_attribute_filters_are_not_relaxed(): +def test_ranking_and_attribute_filters_are_not_relaxed() -> None: attribute = {"type": "attribute_filter", "using": "label/region", "state": {"include": ["EMEA"]}} scores = check_filters( _viz(), @@ -206,7 +217,7 @@ def test_ranking_and_attribute_filters_are_not_relaxed(): @pytest.mark.parametrize("malformed", ["text", {"view": "x"}, {"view": {"dashboard": {"filters": "x"}}}]) -def test_a_malformed_context_is_ignored_not_raised(malformed): +def test_a_malformed_context_is_ignored_not_raised(malformed: object) -> None: assert dashboard_date_filters(malformed) == [] @@ -214,7 +225,7 @@ def test_a_malformed_context_is_ignored_not_raised(malformed): @pytest.mark.parametrize("value", [None, "", "0", "false", "no", "off"]) -def test_flag_off_scores_strictly(monkeypatch, value): +def test_flag_off_scores_strictly(monkeypatch: pytest.MonkeyPatch, value: str | None) -> None: if value is not None: monkeypatch.setenv(_ENV, value) expected, actual = _viz(), _viz({"f": _INHERITED}) @@ -224,14 +235,14 @@ def test_flag_off_scores_strictly(monkeypatch, value): @pytest.mark.parametrize("value", ["1", "true", "yes", "on"]) -def test_flag_on_from_the_environment(monkeypatch, value): +def test_flag_on_from_the_environment(monkeypatch: pytest.MonkeyPatch, value: str | None) -> None: monkeypatch.setenv(_ENV, value) scores = check_filters(_viz(), _viz({"f": _INHERITED}), _TODAY, user_context=_USER_CONTEXT) assert scores.date_ok is True assert scores.date_inherited is True -def test_the_keyword_overrides_the_environment(monkeypatch): +def test_the_keyword_overrides_the_environment(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setenv(_ENV, "1") scores = check_filters( _viz(), _viz({"f": _INHERITED}), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=False @@ -244,7 +255,9 @@ def test_the_keyword_overrides_the_environment(monkeypatch): [{}, {"f": _INHERITED}, {"f": _date(-5, 0)}, {"a": _INHERITED, "b": _date(0, 0, "YEAR")}], ) @pytest.mark.parametrize("expected_filter_by", [{}, {"f": _date(-1, -1, "YEAR")}]) -def test_flag_off_detail_is_byte_identical_with_and_without_user_context(expected_filter_by, actual_filter_by): +def test_flag_off_detail_is_byte_identical_with_and_without_user_context( + expected_filter_by: dict, actual_filter_by: dict +) -> None: expected, actual = [_viz(expected_filter_by)], _viz(actual_filter_by) before, _ = _evaluate_against_candidates(expected, actual, True, _TODAY) after, _ = _evaluate_against_candidates(expected, actual, True, _TODAY, user_context=_USER_CONTEXT) @@ -280,14 +293,14 @@ def _ticket_chat(metric: str = "metric/gross_margin_percent") -> ChatResult: ) -def test_ticket_item_single_shot_fails_with_the_flag_off(): +def test_ticket_item_single_shot_fails_with_the_flag_off() -> None: result = get_evaluator("visualization").evaluate(_ticket_item(), _ticket_chat()) assert result.passed is False assert result.detail["filter_date_score"] is False assert "filter_date_context" not in result.detail -def test_ticket_item_single_shot_passes_with_the_flag_on(monkeypatch): +def test_ticket_item_single_shot_passes_with_the_flag_on(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setenv(_ENV, "1") result = get_evaluator("visualization").evaluate(_ticket_item(), _ticket_chat()) assert result.passed is True @@ -303,7 +316,7 @@ def _agentic_client(chat: ChatResult) -> MagicMock: @pytest.mark.parametrize(("flag", "passes"), [(None, False), ("1", True)]) -def test_ticket_item_agentic_run(monkeypatch, flag, passes): +def test_ticket_item_agentic_run(monkeypatch: pytest.MonkeyPatch, flag: str | None, passes: bool) -> None: if flag is not None: monkeypatch.setenv(_ENV, flag) with patch.object(agentic_visualization, "ChatClient", return_value=_agentic_client(_ticket_chat())): @@ -322,7 +335,7 @@ def test_ticket_item_agentic_run(monkeypatch, flag, passes): assert ev.filter_date_inherited is passes -def test_strict_check_summary_shows_the_inherited_date(monkeypatch): +def test_strict_check_summary_shows_the_inherited_date(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setenv(_ENV, "1") # A wrong metric keeps the item failing, so the summary is printed. chat = _ticket_chat(metric="metric/revenue") @@ -344,7 +357,7 @@ def test_strict_check_summary_shows_the_inherited_date(monkeypatch): assert "Metrics Correct : False" in message -def test_strict_check_summary_is_unchanged_with_the_flag_off(): +def test_strict_check_summary_is_unchanged_with_the_flag_off() -> None: with ( patch.object(agentic_visualization, "ChatClient", return_value=_agentic_client(_ticket_chat())), pytest.raises(VisualizationAssertionError) as exc_info, @@ -363,7 +376,7 @@ def test_strict_check_summary_is_unchanged_with_the_flag_off(): assert "inherited" not in message -def test_failed_checks_leave_out_an_accepted_inherited_date(monkeypatch): +def test_failed_checks_leave_out_an_accepted_inherited_date(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setenv(_ENV, "1") # An extra attribute filter fails the filters; the inherited date filter must not be blamed. chat = ChatResult.model_validate( diff --git a/packages/gooddata-eval/tests/test_eval_error_scores.py b/packages/gooddata-eval/tests/test_eval_error_scores.py index f70fa5762..b5e4b55e5 100644 --- a/packages/gooddata-eval/tests/test_eval_error_scores.py +++ b/packages/gooddata-eval/tests/test_eval_error_scores.py @@ -142,7 +142,7 @@ def _capture(_submit: object, _identity: object, **kwargs: Any) -> None: @pytest.mark.parametrize("name", sorted(_KINDS)) -def test_an_errored_run_gets_eval_error_beside_its_usual_scores(name) -> None: +def test_an_errored_run_gets_eval_error_beside_its_usual_scores(name: str) -> None: labels = RunLabels((Outcome.error("simulated_user_failed"), Outcome.failed("agent_silent"))) ctx = _scores(_KINDS[name], labels) @@ -153,7 +153,7 @@ def test_an_errored_run_gets_eval_error_beside_its_usual_scores(name) -> None: @pytest.mark.parametrize("name", sorted(_KINDS)) -def test_a_chat_error_run_is_scored_as_before_with_no_marker(name) -> None: +def test_a_chat_error_run_is_scored_as_before_with_no_marker(name: str) -> None: ctx = _scores(_KINDS[name]) for run_idx in (0, 1): diff --git a/packages/gooddata-eval/tests/test_outcome.py b/packages/gooddata-eval/tests/test_outcome.py index 4821dbd62..71b75bd6f 100644 --- a/packages/gooddata-eval/tests/test_outcome.py +++ b/packages/gooddata-eval/tests/test_outcome.py @@ -16,7 +16,8 @@ import pytest from gooddata_eval.core.agentic import visualization as agentic_visualization from gooddata_eval.core.agentic._gate import gate_failure_note -from gooddata_eval.core.agentic._outcome import chat_failure_exit, label_runs +from gooddata_eval.core.agentic._outcome import chat_failure_exit, label_runs, mark_unscored_run +from gooddata_eval.core.agentic.document_skill import _judge_error_of from gooddata_eval.core.agentic.visualization import VisualizationAssertionError, evaluate_agentic_visualization from gooddata_eval.core.chat.sse_client import ChatError, TransientChatError, TurnIncompleteError, TurnTimeoutError from gooddata_eval.core.models import AgenticAssertionError, ChatResult, CreatedVisualization, LoopExit @@ -129,7 +130,7 @@ def test_an_errored_item_names_its_most_frequent_reason() -> None: ([PASS, PASS, PASS], "power", "pass", 3), ], ) -def test_the_shadow_verdict(outcomes, gate, verdict, scored) -> None: +def test_the_shadow_verdict(outcomes: list[Outcome], gate: str, verdict: str, scored: int) -> None: assert shadow_verdict(outcomes, gate) == {"verdict": verdict, "runs_scored": scored, "runs_total": len(outcomes)} @@ -157,6 +158,50 @@ def test_label_runs_reads_each_runs_exit_and_judge() -> None: assert labels.shadow("power") == {"verdict": "fail", "runs_scored": 2, "runs_total": 4} +def test_a_kind_can_name_its_own_judge_error() -> None: + run = _Run(False, LoopExit.SUCCESS, judge_error="takeaway request failed") + assert label_runs([run], passed=lambda r: r.ok).outcomes == (JUDGE,) + labels = label_runs([run], passed=lambda r: r.ok, judge_error=lambda r: None) + assert labels.outcomes == (Outcome.failed("success"),) + + +class _Evaluation: + def __init__(self, judge_error: str | None, fill_error: str | None) -> None: + self.judge_error = judge_error + self.fill_error = fill_error + + +class _DocRun: + def __init__(self, judge_error: str | None, fill_error: str | None) -> None: + self.evaluation = _Evaluation(judge_error, fill_error) + self.exit_reason = None + + +def test_a_document_run_whose_takeaway_request_failed_is_a_failure_not_a_judge_error() -> None: + request_failed = _DocRun(judge_error=None, fill_error="503 from the takeaway endpoint") + judge_failed = _DocRun(judge_error="empty body twice", fill_error=None) + labels = label_runs([request_failed, judge_failed], passed=lambda r: False, judge_error=_judge_error_of) + assert labels.outcomes == (Outcome.failed(None), JUDGE) + + +def test_an_unscored_run_gets_the_marker_only_when_the_harness_failed() -> None: + ctx = MagicMock() + mark_unscored_run(ctx, 0, "conv-1", JUDGE) + ctx.export_unscored.assert_not_called() + ctx.score.assert_called_once() + assert ctx.score.call_args.kwargs == { + "name": "eval_error", + "value": 1, + "data_type": "NUMERIC", + "comment": "judge_failed", + } + + ctx = MagicMock() + mark_unscored_run(ctx, 1, "conv-2", None) + ctx.export_unscored.assert_called_once_with(1, "conv-2") + ctx.score.assert_not_called() + + # ── end to end through the visualization kind: labels, never a new verdict ─── @@ -256,7 +301,7 @@ def test_an_item_whose_every_run_errored_is_labelled_and_still_fails() -> None: (ChatError("SSE stream error"), "chat_error"), ], ) -def test_a_chat_failure_is_a_plain_failure(exc, exit_value) -> None: +def test_a_chat_failure_is_a_plain_failure(exc: BaseException, exit_value: str) -> None: with pytest.raises(VisualizationAssertionError) as exc_info: _evaluate([exc, _viz_turn(), _viz_turn()]) From a190a5c13c0df57e4aded1356ba343209e91f54f Mon Sep 17 00:00:00 2001 From: "my.nguyen" <my.nguyen@gooddata.com> Date: Tue, 17 Nov 2026 14:28:18 +0700 Subject: [PATCH 3/5] docs(gooddata-eval): restore README sections dropped by the relabel rework The phase-1 rewrite of the README replaced too much text: it removed the latency breakdown, `gd-eval report`, `models`, `generate`, the dataset format and other sections that this PR never meant to touch. Restore them; the PR now only adds the --accept-context-date-filter row, the failing-runs paragraph and the harness-error section. jira: QA-30213 risk: low Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- packages/gooddata-eval/README.md | 637 +++++++++++++++++++++++++++++++ 1 file changed, 637 insertions(+) diff --git a/packages/gooddata-eval/README.md b/packages/gooddata-eval/README.md index 9f4133d4a..905fd5d97 100644 --- a/packages/gooddata-eval/README.md +++ b/packages/gooddata-eval/README.md @@ -322,6 +322,643 @@ turns' exits but does not label them yet. on `exit_reason == "chat_error"` see fewer runs. The item budget (`--item-timeout`) runs from conversation creation, so an `agent_timeout` from it can include the simulated user's replies and the chat retries' backoff. +Each item additionally carries a per-phase breakdown: + +```json +"latency_breakdown_s": { + "agent_s": 4.02, // GoodData's own response time — the system under test + "judge_s": 1.31, // LLM-as-judge scoring, post-hoc + "simulated_user_s": 0.0, // our simulated user composing the next turn (multi-turn kinds) + "langfuse_s": 5.70 // trace lookup + score writing, off the critical path +} +``` + +Every item reports `runs_ungraded` beside `runs_passed`: runs the agent answered but the LLM judge returned +nothing readable for. Agentic items also list them as `unscored_runs` / `judge_errors` in their `detail` (a document item whose takeaway-endpoint request failed +lists that failure under `request_errors` instead), and a +`dashboard_summary` item carries `ungraded_criteria`. Such a run — or, for `dashboard_summary`, such a +criterion — is excluded from pass@K and from the quality score rather than counted as a failure: scoring it 0 +would be indistinguishable from the judge genuinely failing the answer, which is the confusion +`JudgeResponseError` exists to end. `pass@K` still holds on the runs that *were* graded, so an item can pass with +`runs_ungraded` set; `pass^K` cannot, because a run nobody graded leaves "all K passed" unverified. For +`dashboard_summary` an ungraded `must_include` / `must_not_include` criterion likewise cannot carry a pass — +"the judge could not tell" is not evidence the fact is present — while an ungraded `rubric` line only narrows +the quality score. When *no* run could be graded (for `dashboard_summary`: no gating criterion on any run) the +item errors instead of reporting failures. A non-zero count means pass@K was computed over fewer runs than +`--runs` asked for, so treat the result as weaker evidence and check the judge (`GD_EVAL_JUDGE_DIAGNOSTICS=1`, +or raise `JUDGE_MAX_COMPLETION_TOKENS` if the cause is `finish_reason=length`). The console says so in `Notes` +(`1 run(s) ungraded`, `2 criterion(s) ungraded`). + +`agent_s` + `judge_s` + `simulated_user_s` are the instrumented parts of the item's `latency_s`; they do not add +up to it exactly, because `latency_s` is wall-clock around the whole item and also covers the conversation +create/delete round trips, SDK construction and any cleanup. `langfuse_s` sits **beside** `latency_s`, never +inside it, because trace linking runs outside every item's critical path (see above) — summing all four would +re-inflate exactly what that design removes. + +A phase that a kind does not have reports `0.0` rather than an invented number, so read the zeroes as "not +applicable here", not "instant". Today: + +| Field | Populated by | +|---|---| +| `agent_s` | `agentic_general_question`, `agentic_metric_skill` | +| `judge_s` | `agentic_general_question` only — `agentic_metric_skill` compares MAQL by string, it has no LLM judge | +| `simulated_user_s` | `agentic_metric_skill` only — `agentic_general_question` is single-turn, it has no simulated user | +| `langfuse_s` | every agentic kind, but only on the `gd-eval` path and only when Langfuse credentials are present | + +The other six agentic kinds report `0.0` for the first three. Trace linking itself happens whenever +`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported, with or without `--langfuse`, because each +`evaluate_agentic_*` falls back to `try_make_langfuse_client()`. But its *duration* is measured by the CLI +runner rather than by `evaluate_agentic_*`, so a direct library caller sees `langfuse_s: 0.0` even though its +linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogether. + +### Latency: single-shot and multi-turn + +`avg_latency_s` is one number over a mixed population. The report instead keeps two kinds of item apart, by +their eval kind, and states latency for each: + +- **Single-shot** — kinds that send exactly one chat message per run: every non-agentic kind, plus + `agentic_general_question`, `agentic_guardrail` and `agentic_search`. +- **Multi-turn** — kinds that drive a simulated user or fixture turns: `agentic_visualization` (and + `vis_agentic`), `agentic_metric_skill`, `agentic_alert_skill`, `agentic_dashboard_skill`, + `agentic_document_skill`, `agentic_kda_skill`, `agentic_anomaly_detection`, `agentic_what_if` and + `agentic_conversation`. + +```json +"summary": { + "by_mode": { + "single_shot": {"total": 12, "passed": 10, + "traces_s": {"n": 24, "mean": 5.39, "trimmed_mean": 5.36, "min": 3.88, "p50": 5.33, "p70": 6.23, "p90": 6.82, "p99": 7.18, "max": 7.22}}, + "multi_turn": {"total": 6, "passed": 4, + "traces_s": {"n": 45, "mean": 6.91, "trimmed_mean": 6.84, "min": 2.78, "p50": 6.52, "p70": 8.03, "p90": 10.79, "p99": 13.57, "max": 14.25}, + "runs_s": {"n": 15, "mean": 23.55, "trimmed_mean": 23.31, "min": 12.64, "p50": 21.23, "p70": 28.52, "p90": 33.02, "p99": 36.94, "max": 37.51}} + } +} +``` + +`by_mode` is in each run's `summary` and in its `comparison` entry. `total` and `passed` count the evaluated +items of that mode (skipped ones belong to neither). Every distribution is `null` when nothing was timed. +Each item also states its `mode`, its `trace_latencies_s` and its `run_latencies_s`. + +- **Per trace is the main figure.** gen-ai starts one trace per chat message, so a trace is one chat turn. + Its latency is that turn's response time, from sending the message to the end of its stream + (`turn_wall_clock_sec`), recorded where the chat client completes the turn. It holds none of the LLM judge, + the simulated user or conversation create/delete. `dashboard_summary` is timed the same way around its one + request. A turn cut off by its turn or item deadline, or one that ended without a final answer (the agent's + iteration or token limit, a content filter), still counts, with the time it ran for: those are the slowest + turns, and the run that scored them is in the per-run figures too. A turn that ended in any other error (an + error event, or a 503 that outlasted its retries) does not: the time is that of a failed request, not a + response, and would only add a fast outlier. On the single-shot path such a turn ends its item as an error, so + it has no per-run figure either. An item that errored outside its evaluator's own failure reporting + contributes neither. +- **Single-shot** runs are exactly one trace, so `traces_s` is also the latency of a run. +- **Multi-turn** runs span several traces, so they get both views: `traces_s` over every turn of every + conversation (comparable with single-shot), and `runs_s`, the wall clock of each whole conversation from just + before its first message to the end of its scoring — every turn, simulated-user reply and judge call, but not + conversation create/delete. `agentic_conversation` drives its fixture once and contributes one run per item. + The two single-shot and multi-turn populations are never mixed. +- **Percentiles** use linear interpolation between closest ranks (NumPy's default), so `p50` is the median and + the report carries no separate median field. On a small sample the high percentiles sit at or near the + maximum: P99 over fewer than 100 samples is effectively the slowest one, which is why `n` always travels with it. +- **Trimmed mean** (`trimmed_mean`) is the average of the middle 80%: the fastest 10% and the slowest 10% of the + samples are dropped from each end (`n * 10 // 100` of them, so 25 samples drop two each) before averaging. It is + not the median, which is one middle value (`p50`). It is `null` below 10 samples, where there is nothing to + trim and the figure would only repeat `mean`. +- **Older reports** have no `by_mode`. The HTML report says so and nothing is backfilled. + +`avg_latency_s` and the other existing fields are unchanged. The console prints each run's pass counts per +mode and one `Latency, <population> (n=...)` line per distribution beneath its summary line, and Trace +P50/P90/P99 columns (single-shot) in the multi-model comparison. + +--- + +## `gd-eval report` + +Turns JSON report(s) into one HTML file you can actually navigate. No server, no +credentials, no external assets — it opens over `file://`, attaches to a Jira issue and +survives a Slack thread. It follows GoodData's product look (colours, type, status chips). The faces are Inter, with +Funnel Display for headings, named but never loaded, so a viewer without them gets the system sans and the file stays +self-contained. + +```bash +# one run +gd-eval report results.json -o report.html + +# several runs side by side -- each file becomes its own column +gd-eval report aug-21.json sep-07.json -o comparison.html --title "H200 regression check" + +# customer-safe +gd-eval report results.json -o customer.html --redact +``` + +| Flag | Description | +|---|---| +| `-o, --out PATH` | Where to write the HTML. Required. | +| `--title TEXT` | Title shown in the report header. | +| `--redact` | Drop conversation/response ids and raw reasoning, and rename models to `Model A`, `Model B`, … A model keeps its letter across all its runs, which are then numbered (`Run 1 · Model A`, `Run 2 · Model A`). Pass rate, per-item results, questions and latency survive. | + +The report is a *view* over the JSON. Unfiltered, every figure on it is read from the report, so it cannot +disagree with the console or the JSON. Narrow the selection with a filter and the figures are recomputed in the +page from the selected items' own records, by the same rules gd-eval uses. The page reads top to bottom as a +document, in plain words: a *test case* is one prompt given to the assistant, an *attempt* is one go at it, and the +two formats are *single-question* tests (one answer, no follow-up) and *conversation* tests (a back-and-forth with a +simulated user), which are `single_shot` and `multi_turn` in the JSON. + +- **A masthead** with the report's title (`--title`, or "Run history", "Run comparison", "Model comparison" or + "Run results" plus the model and workspace), when it was generated in the reader's own time zone, and a **Run setup** tooltip listing what + is the same in every run (model, provider, workspace, dataset, test cases, attempts per test case, pass rule) and + what differs. Results from different setups are not directly comparable, so a caution icon on a latency row and the + *New* / *Removed* marks in the test case table say where two runs stopped being alike. +- **View by** — a dropdown above the filters that lays the same report out two ways. **Date** is the layout the + rest of this list describes: a column per run, oldest to latest by start time, with change lines, sparklines and the latency line + chart. **Model** compares language models instead: a column per model, ranked best first by the console's winner + rule (pass rate, then quality, then the quicker average), with no time axis. Where a model has several runs its + latest stands for it, so each model is read at its most recent state. The Run filter becomes a Model filter, the hint + beside the dropdown says what is being compared (and names the dates when the compared runs differ in date), and the + Run setup tooltip does not flag the model itself as a difference. Every report opens on Date, including one that + compares several models (`gd-eval run --model a --model b --html`); choose Model to compare them. A single run has + nothing to switch between, so the dropdown is hidden. In the Model view the page reads, top to bottom: + - **Results** — one row per model, best pass rate first, with a bar of how its evaluated test cases ended (passed + green, failed red, could not be graded orange), so the green part is the pass rate beside it, and the **Median + latency** of one attempt (half the attempts were faster, half slower). A list under the table says what each term means. The counts and the + caution icon (a model evaluated on different test cases than the one with the most) are the date view's Outcomes + table. There are no headline cards by model: the table is the first thing on the page. + - **Quality against speed** — each model as a point: the share of test cases it passed against the median time of + one attempt, the same pass rate as the Results table and the same median as the latency tables. Up and to the left is better. Time runs from zero; + the pass rate uses the range the models span. It needs two models with timings. + - **By category** — a heat map of the pass rate per category and model, the best model in each row marked, the + categories where the models differ most first. Every cell prints its share and its counts, so the shade is never the + only signal. This is where a model that wins overall but loses on one category shows. + - **Consistency across attempts** — per model, the share of test cases that passed on at least one attempt (pass@K, + the light bar) beside the share that passed on every attempt (pass^K, the dark bar), and how many passed only some. + A long gap is a model that can do the task but not reliably. It appears only for a report that recorded + `pass_power_k` and attempted a test case more than once. + - **Latency sections** — the same four percentiles, one row per model, quickest first, with a bar per figure drawn + from zero beside the numbers. The quickest and slowest model in each column are marked as in the date view. + - **Test cases** — one result column per model in ranked order, a response-time column per model, and an **Across + models** column in place of *Since previous run*: *Mixed* (some models passed and some did not; hover to see who), + *All pass* or *All fail*. A skipped or ungraded test case is left out of the verdict. The table opens on the + *Mixed* ones, the test cases where the models differ. +- **Filters in two tiers** — the **Filter the report** bar at the top (category, format, result and run) decides what + the whole page covers: the Results cards, the outcomes table, the latency tables and the test case table all follow it, so the runs stay + comparable. The Test cases section repeats those four and adds the search box and the expression; they narrow the + table only, inside whatever the top bar allows, and only offer what the top bar leaves. Category, result and run are + attribute filters like the product's: a search box, an All row, the values with an **Only** shortcut and a count, + then Cancel and Apply. Changes are staged in the popup and take effect on Apply; closing it any other way discards + them. Nothing ticked means all. The run filter chooses which runs are shown; a result matches a test case when any of + the runs in play has it (so "Failed" over previous and latest finds every test case that broke in either), and the + expression is evaluated once per run in play with the same any-run rule. A filtered page states how many test cases + it covers, and each tier has its own Reset. +- **Results** — three headline cards for the latest run: test cases passed (of those evaluated: skipped ones are not + counted, as in the console's percentage), the average quality score and the median time per attempt. That last one + pools every format on the page and counts whole attempts, grading included, so it matches neither per-response latency table + below it; its tooltip says which formats it covers. With an earlier run each card carries one change line in words, "down from 86% on Sep 28", rather + than a signed number. The line is green when that direction is good and red when it is not, with a chevron and + screen-reader text (*Better* / *Worse*) so colour is not the only signal. From three runs each card also has a + sparkline of every run: hover a point for its value, click it to compare the latest run with that one instead of the + previous. The dot being compared with is drawn white, and a key under the cards says so (latest run, compared with, + other runs) and that a click on any other dot moves the comparison; it shows only when a sparkline does. A pick that + a filter has removed, or that has become the latest run, falls back to the previous run. The + pass-rate card links to the test case table to see *which* ones changed; the split by format lives in that table's + filters. +- **Outcomes** — how every test case ended, as a table with one row per run, newest first, in the same layout as the + latency tables. A run is identified by three columns: the **Run date**, the day and time it started (`summary.started_at`, + shown in the reader's own time zone, the UTC stamp on hover; a report written before it existed shows the date its file is + named for, without a time), and the **Run name**, the name the run was given, as given, so a run named `2026-09-28` reads + `2026-09-28` here although the charts style it as "Sep 28", and the **Model** it evaluated, its provider on hover. Then **Passed**, **Failed**, **Could not be graded** (an error left it without a verdict) and + **Skipped** (not run), then the **Pass rate**. The four are the values of the Result filter, and the pass rate is + the card's: passed out of the evaluated test cases, skipped ones left out, as in the console. The highest pass rate + is marked green with an up chevron and the lowest red with a down chevron, and the latest run is marked in lime. A + caution icon flags a run evaluated over a different number of test cases than the one before, because its pass + rate is then not a like-for-like change. The counts are read from the run's `summary` (`passed`, `failed`, + `errored`, `skipped`); a filtered view recounts them from the selected test cases, and a report written before + `summary.failed` existed shows what the other three leave. +- **Latency sections** — **Single-question tests · response time** first, then **Conversation tests**, which shows + both per response (each assistant reply, one gen-ai trace) and per conversation (start to finish, with the + simulated user and the judge). Each opens with a **line chart** of the median (P50), P70, P90 and P99 across the + runs, oldest on the left and the latest on the right, drawn from zero so line heights compare honestly; the lines + share one blue that darkens towards the tail. Hover or focus a run for its crosshair and every figure, in a tooltip + that says how many responses it rests on. A line needs two runs with timings, so with one run the chart is left + out. End labels name each line and give its latest value, and are dropped when two would touch. Under the chart a + table has one row per run, newest first, identified by the same **Run date**, **Run name** and **Model** columns as the outcomes table, with the same four figures. The quickest run in each column is marked + green and the slowest red, each with a chevron (down for least time, up for most) so colour is not the only + signal, and the latest run is marked in lime. A caution icon flags a run timed on a different number of samples + than the one before. Unfiltered they are read from `by_mode`; a filtered view recomputes them in the page by the + same rules. The fastest and slowest single response, the average and the trimmed average stay in the JSON (P99 + over a few dozen samples sits close to the slowest one, which is why `n` travels with it in the tooltip and + the cell hint). Seconds read with one decimal and no redundant ".0" (`3s`, `4.6s`). +- **A test case table**, newest first: the latest run's result, a **Since previous run** column, then the earlier runs + going back in time (click any result to open the drawer on that run). Each run's column is headed by its run date over + its run name, one to a line (hover the heading for the full run label); by model the columns are headed by the model. The change column reads *Regressed* (passed in + the previous run, not in the latest), *Fixed*, *Flaky* (flipped between pass and fail more than once), *New* + (in no earlier run) or *Removed* (in an earlier run, gone from the latest). A test case skipped in the latest run + has no label, since the change would describe earlier runs. With several runs the table opens on the regressions. Two time columns + keep the populations apart: **Response time** is the median of the test case's assistant responses, and **Time per + attempt** is the mean wall-clock of a whole attempt (the responses plus the judge and, for conversations, the + simulated user). With more than two runs shown they follow the latest run only. Hover a value to see every sample + behind it; the drawer labels its two latency chips the same way. A report written before per-trace timing has no + response figure and shows `avg_latency_s` as the attempt figure. +- **An expression filter**, folded under *Advanced: filter by a check expression* in the Test cases section, for + cross-cutting questions the fixed filters can't anticipate, e.g. `d.filter_ranking_score === false` or + `d.expected_metric_uris.length > 1 && !d.metrics_correct`. Available variables: + `d` (the checks of the run being read, its `detail`), `it` (that run's result), `i` (the row, `i.per[label]` for any + run), `q` (question), `kind` (category id). +- **The conversation**, when the test case ran the agentic multi-turn path — every turn in + order, with the simulated user marked apart from a real question, so you can see + whether the agent got there or was handed the answer. +- **A per-test-case drawer** — checks as pass/fail chips, expected vs actual side by side, + the latency timeline, conversation/response ids. +- **A latency timeline** from `detail.latency_breakdown`, in execution order, one bar per + step. Every step has a chevron that opens its full record in place, joined by `index`: the + paragraph a reasoning step was summarised from, or a tool call's arguments and result from + `detail.tool_calls`. A reasoning paragraph's markdown `**title**` is shown as bold text, + not as asterisks. Any number of steps can be open at once, and the chevron above the list + opens or closes all of them. The reasoning therefore lives on the timeline; a separate + Reasoning section appears only for paragraphs no step points at (or when a run recorded no + timeline). +- **A print stylesheet** that drops the filters and the drawer, so "Save as PDF" gives a clean page. + +A report that merges several files names each run after its file, and a file named for its date (`2026-09-28.json`) +reads as "Sep 28". A model is read from the run itself (`model` and the provider), never from the file name, so files +named for their model work as well as files named for their date. In the Date view the page treats the runs as one +model followed over time; when a report holds several models it adds the model to each run's name and points at +**Model** (see above) for the comparison. + +`--redact` additionally drops `transcript` and `tool_calls`: the exchange shows that the +simulated user is primed with the expected output, and a tool result carries +semantic-layer internals and real query rows. The turn count and the timeline shape +survive. + +Passing several files keyed by file name is the whole run-over-run mechanism: no +database, no run registry, just the JSON files you already have on disk. + +--- + +## `gd-eval models` + +List all LLM providers and their models in the org. Marks the active model +for a workspace when `--workspace` is given: + +```bash +gd-eval models \ + --host https://your.gooddata.cloud \ + --workspace ecommerce_demo +``` + +``` +┃ Provider ┃ Provider ID ┃ Model ID ┃ Family ┃ Active ┃ +│ Foundry4o │ foundry_… │ gpt-5.2 │ OPENAI │ ◀ active │ +│ │ │ gpt-4o │ OPENAI │ │ +│ HN_Anthropic │ hn_anthr_… │ claude-opus-4-7 │ ANTHROPIC │ │ +``` + +--- + +## `gd-eval generate` + +Reverse-engineers a `visualization` dataset out of the charts a customer has already +built, so you get eval questions without hand-authoring any. Reads the workspace's +declarative analytics model (read-only), translates each visible insight's buckets, +sorts and filters into an `expected_output.visualization` spec, then asks an LLM to +write the analyst question that chart answers. Because `expected_output` is copied from +a live object rather than authored, every question is grounded in the real LDM by +construction — the LLM only writes English. + +**Setup:** host + token (read access to the workspace), and `OPENAI_API_KEY` plus the +`llm-judge` extra for the phrasing step (`uv add 'gooddata-eval[llm-judge]'`; skip both +with `--no-phrase`). + +```bash +export GOODDATA_TOKEN='your-api-token' + +# 1. see what a workspace yields before writing anything +gd-eval generate \ + --host https://your.gooddata.cloud \ + --workspace ecommerce_demo \ + --dataset-name ecommerce \ + --dry-run + +# 2. generate, phrase, validate, and export +gd-eval generate \ + --host https://your.gooddata.cloud \ + --workspace ecommerce_demo \ + --dataset-name ecommerce \ + --dashboard dash_1_returns \ + --out ./my-dataset \ + --langfuse-out out/langfuse-dataset.json + +# 3. run it +gd-eval run --host … --workspace ecommerce_demo --dataset ./my-dataset --model gpt-5.2 +``` + +`--workspace` is where insights are read from; `--dataset-name` is the `dataset_name` +written into every item (and the default output folder). + +| Flag | Effect | +|---|---| +| `--dashboard <id>` | restrict to insights on that dashboard (repeatable); default is the whole workspace | +| `--out <dir>` | output folder (default `./<dataset-name>`); this is what `gd-eval run --dataset` reads | +| `--snapshot-out` / `--snapshot-in` | save/replay the fetched model — replay needs no host, token, or network | +| `--langfuse-out <file>` | also write a Langfuse-importable dataset JSON | +| `--id-prefix` | prefix exported Langfuse item ids (they're unique per *project*, so re-importing an item under its original id is a 409) | +| `--no-phrase` | skip the LLM; emit mechanical `Show <title>` questions | +| `--phrase-model` | OpenAI model for phrasing (default `gpt-4o`) | +| `--no-viz-type` | always blank the expected chart type | +| `--enrich-ranked <N>` | additionally derive up to N ranked questions (see below); default 0 (off) | +| `--skip-ambiguous` | drop items naming something the model carries more than once; reported either way | +| `--min-questions` / `--min-shapes` / `--min-filtered` | quality gate, default 15, 3 and 1 | + +### Ranked questions (`--enrich-ranked`) + +Analysts sort in Analytical Designer and save the chart without persisting the sort, so +`sort_by`/`ranking_filter` coverage is near zero on most real models — the eval can +punish a spurious ranking but never confirm the agent builds a required one. +`--enrich-ranked N` fills that gap by *deriving* ranked items from the specs already +extracted. Adding a limit or a sort to a definition that executes cannot make it +unanswerable, and "the top 3 X by Y" has exactly one correct spec, so a derived item is +less ambiguous to grade than the insight it came from. + +The budget is spent best-grounded first: + +1. **Insights whose own title promised a ranking their definition never implemented** — + "Top Returned Reasons" saved with `sorts: []`. The direction comes from the title + (`highest`/`most`/`largest` vs `lowest`/`least`/`worst`) and the N too when it states + one; a title naming both ends names neither and is still skipped. +2. **Ranking filters added to a plain breakdown** — one metric, one non-date dimension, + no existing sort. N follows the dimension's element count, so a top-5 over six values + is never emitted. +3. **Sort-only variants**, which order without limiting. + +Eligibility is deliberately narrow: two metrics leave "top 3 by what?" unanswered, a +second dimension leaves the N ambiguous between the pair and within a group, and a date +dimension turns the result into "top 3 months", which nobody asks. Variants are +deduplicated by resolved definition — differently-titled insights over one metric and +dimension would otherwise produce the same question twice — and bases are taken +round-robin by metric so one popular metric cannot become a third of the corpus. + +Derived items carry `derived_from` (the insight id) and `derived_basis` (`title` when a +human's chart title asked for the ranking, `shape` when this generator chose to add +one), so a pass rate over each can be computed separately. + +### Items that cannot say what they mean + +Two classes of question are unwinnable however well the agent behaves, and both are +reported: + +- **A name the model carries more than once.** One workspace has six labels all titled + "Product Title"; a question naming one cannot say which is meant, and a perfect chart + over the wrong one scores zero. `--skip-ambiguous` drops them; the count and the + offending names are printed either way. +- **A date granularity's cyclical twin.** `MONTH` walks consecutive calendar months, + `MONTH_OF_YEAR` stacks every January together. Date dimensions are therefore briefed + by what they do ("one point per calendar month over time, not month-of-year") and the + writer is told to say it in natural words while keeping the date dataset's name — + never as a label id in prose ("Order Created At - Month"). + +**The question must never contradict its own expected output.** Four rules enforce that: + +- The writer is briefed on buckets, sorts and filters only — never the insight title, + and never the chart type. Titles routinely describe intent the definition doesn't + implement ("Products by Most Items Sold" over `sorts: []`). +- Every generated question is checked against its spec, and any hit is a hard error: + ranking words (`top`, `most`, `highest`, …) require a real sort or ranking filter; + filter words (`only`, `last quarter`, `in 2025`, …) require a real date or attribute + filter; a breakdown clause requires a non-empty `view_by`/`segment_by` and vice versa; + a metric may never be broken down by itself; and no template residue (`breakdown + dimension`, `{…}`) may survive. A violation is fed back once for a rewrite, then + dropped — and a drop fails the run. +- The writer's rules are built per insight, so an insight with no `view_by` is never + asked to name a breakdown at all. +- `type` is set only when the question actually names a chart form. An insight's + `visualizationUrl` records what a human clicked, not what the question constrains — + with one exception: a chart with no breakdown *must* name its form ("as a KPI", "as a + single number"). Without it the agent reads a bare "Show me Gross Revenue" as a metric + lookup, activates only its search skill and builds nothing. + +Everything the writer sees is a display name (`Spend Amount`, `Merchant Name`), never a +raw URI, so questions read like a person wrote them. + +**What it won't do.** Insights it can't express without guessing are skipped with a +printed reason, never approximated: derived (arithmetic/PoP) measures, measure-level +filters, `uris`-form attribute filters, unmapped chart types, hidden objects, and +insights whose title promises behaviour their definition lacks (though `--enrich-ranked` +implements a promised *ranking* rather than discarding it). If too few survive, the +quality gate fails the run rather than fabricating items to hit the minimum — point at +more dashboards, or lower `--min-questions`. + +Every written item is validated as a `DatasetItem` with a scorable AAC visualization +before the command reports success. + +--- + +## Dataset format + +A dataset is a folder of `.json` files, one per question: + +```json +{ + "id": "stable-unique-id", + "dataset_name": "my_dataset", + "test_kind": "visualization", + "question": "Show revenue by quarter", + "expected_output": { } +} +``` + +Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`, +`search_tool`, `general_question`, `guardrail`, `dashboard_summary`, `document_takeaway`. + +### `visualization` items + +The `expected_output` lists the candidate charts in the AAC format, and the item passes when +the agent's chart matches one of them. Keys beside the candidates add item-level checks: + +```json +{ + "id": "sankey-001", + "dataset_name": "agent_visualization_extended_charts", + "test_kind": "visualization", + "question": "Show how net sales flow from product category to customer country.", + "expected_output": { + "visualization": { + "type": "sankey_chart", + "query": {"fields": {"m": {"using": "metric/net_sales"}, "c": {"using": "label/product_category"}, + "d": {"using": "label/customer_country"}}}, + "metrics": ["m"], + "from": "c", + "to": "d", + "buckets_strict": true + }, + "requires_visualization_part": true + } +} +``` + +Item-level keys, all optional except the candidates: + +- `visualization` — one candidate chart, or a list of them. `expected_outputs` + (`[{"visualization": {...}}, ...]`) is the same list in the agentic fixture form. +- `requires_execution` — the agent must also run the chart it built. +- `expects_visualization: false` — the agent must answer without a chart, for example by asking + for a part the request leaves out. The item then lists no candidates. +- `visualization_optional: true` — either an answer without a chart, or a chart that matches a + candidate, is right. +- `answer_must_include` / `answer_must_not_match` — regular expressions, matched without regard + to case against the final answer. Every `answer_must_include` pattern must match, and no + `answer_must_not_match` pattern may. +- `format_intents` — per metric uri, how the metric's numbers must read, for example + `{"metric/net_sales": {"divide_by": 1000000, "decimals": 1, "suffix": "M"}}`. +- `requires_visualization_part` — the chart must come back as a part of the answer, not only as + the arguments of the chart tool call. + +Switches on a candidate chart: + +- `buckets_strict` — compare bucket by bucket, not only the union of the buckets, so swapped + heatmap rows and columns or a reversed sankey fail. +- `metrics_ordered` — compare the metrics by position, for bullet roles and bubble axes. + +Any other key, at item level or on a candidate chart, makes the item error before any request +is sent, because a misspelled key would otherwise be dropped and its check would never run: + +``` +ValueError: the expectation states ['answer_must_inlcude'], which is not one of ['answer_must_include', ...] +ValueError: a candidate chart states ['bucket_strict'], which the chart model does not read +``` + +### `dashboard_summary` items + +Summary items call the dedicated summary endpoint +(`POST /api/v1/ai/workspaces/{ws}/summary`) instead of the chat endpoint, so +they carry an extra `summary_input` block, and the `expected_output` is a +**rubric** rather than an exact answer (summaries are free text): + +```json +{ + "id": "summary-001", + "dataset_name": "summary_pilot", + "test_kind": "dashboard_summary", + "question": "Summarize the Sales Overview dashboard.", + "summary_input": { + "dashboard_id": "sales_overview" + }, + "expected_output": { + "must_include": ["States the overall revenue trend", "Identifies the top segment"], + "must_not_include": ["Numbers or segments not present in the visualizations"], + "rubric": ["Reads as a coherent business summary"] + } +} +``` + +`summary_input` requires only `dashboard_id` (the endpoint summarizes the whole +dashboard). Optional fields narrow the scope: `visualizations` (list of ids), +`filter_context` (AFM filters), `tab_id`, and `format_hint`. + +The `expected_output` rubric: + +- `must_include` — facts a good summary must contain; **all** must pass for the item to pass. +- `must_not_include` — hallucination/accuracy guards; **any** violation fails the item. +- `rubric` — soft quality dimensions; they affect `quality_score` but do not gate pass/fail. + +Each criterion is scored independently by the LLM judge, so `quality_score` +is the fraction of satisfied criteria. + +### `document_takeaway` items + +Takeaway items call the AI Publisher takeaway endpoint +(`POST /api/v1/ai/workspaces/{ws}/takeaway`) with a `takeaway_input` block, authored in +snake_case and sent as the endpoint's camelCase body. In a Langfuse dataset the block goes +in the item's `metadata` (or its `input` object), and `input` can stay a plain case label; +an item carrying `takeaway_input` is a `document_takeaway` item unless it declares another +`test_kind`. The endpoint needs the `enableGenAiReportTakeaways` feature flag. + +```json +{ + "id": "takeaway-001", + "dataset_name": "agent_document_takeaway", + "test_kind": "document_takeaway", + "question": "P01 default format, Sales page, no instructions", + "takeaway_input": { + "visualizations": ["<visualization id>", {"visualization_id": "<id>", "filters": []}], + "filter_context": [], + "instructions": {"document": null, "slot": null, "format_hint": null}, + "max_characters": 300 + }, + "expected_output": { + "no_data": false, + "format": {"no_lists": false}, + "must_include": ["States one finding that connects at least two of the charts"], + "must_not_include": [], + "rubric": [] + } +} +``` + +`takeaway_input` requires `visualizations`; unknown keys are rejected. Every +`expected_output` key is optional: + +- `no_data` — whether the endpoint must answer `noData`; when either side says no data, + nothing else is graded. +- `format` — overrides of the deterministic checks, which gate pass/fail: `bold_headline`, + `max_headline_words` (10), `max_sentences` (2), `max_words` (50), `single_paragraph`, + `no_lists`, `no_headings`. `false` or `null` switches a check off; unknown keys are + rejected. `length` (`max_characters`), `no_ids` and `no_meta` always apply. +- `default_rules` — `false` drops the default judge criteria (one finding; no executive + summary, chart walk-through, background opener, figures, dates, causes the data cannot + show, chart description). +- `must_include`, `must_not_include`, `rubric` — the item's own judge criteria, graded like + `dashboard_summary`. + +`word_count` and `headline_word_count` are reported in the item detail. The default judge +criteria are calibrated with `make calibrate-takeaway` (needs `OPENAI_API_KEY`); re-run it +after changing their wording. + +The client retries 429/5xx answers and connection failures. A request error that survives +the retries is handled differently by the two entry points: `gd-eval run` aborts the item +with that error, whatever `--runs` is, while the CI entry point +(`evaluate_document_takeaway`) records it as an errored run, keeps the item's other runs, +and raises only when no run completed. A read timeout is not retried, and the CI entry +point records it as an errored run too. Any other 4xx and an unsupported URL scheme are +never retried and fail the item in both. + +### Items asked in a dashboard context (`user_context`) + +An item can be asked the way a user asks from an open dashboard or an attached widget. +Its `user_context` is sent verbatim as `userContext` on every chat message of the item, +follow-up and clarification messages included — the server treats a message without one +as a cleared context. Requires the `enableAiContextSetup` feature flag on the target +organization — without it the server ignores the dashboard view. Applies to chat items +only; `dashboard_summary` and `document_takeaway` items carry their scope in +`summary_input` / `takeaway_input` instead. No +`userContext` key is sent at all when neither the item nor, for `agentic_conversation`, +the current turn sets one. + +```json +{ + "id": "ctx-001", + "dataset_name": "my_dataset_ctx", + "test_kind": "general_question", + "question": "Which dashboard am I looking at, and what does it cover?", + "user_context": { + "view": { + "dashboard": { + "id": "sales_overview", + "title": "Sales Overview", + "widgets": [ + {"widgetType": "insight", "widgetId": "w1", "title": "Revenue by Month", "visualizationId": "revenue_by_month"} + ] + } + } + }, + "expected_output": "Names the Sales Overview dashboard and summarizes its charts." +} +``` + +The schema belongs to the AI chat API (`UserContext`: `view.dashboard`, +`referencedObjects`, `activeObject`), so it is not validated here beyond being an object. +In a Langfuse dataset, put it in the item `metadata` (or in the `input` object) under +`user_context`. An object in `input` wins and `metadata` is then not read; a missing or +`null` value in `input` falls back to `metadata`. The value used must be an object or `null`; +anything else fails the dataset load. + #### Date filter carried over from the dashboard (opt-in) A user who asks for a new chart from an open dashboard without naming a period may get From 14b26a59d86b3bf8fa264f7b96e453738a717f26 Mon Sep 17 00:00:00 2001 From: "my.nguyen" <my.nguyen@gooddata.com> Date: Tue, 17 Nov 2026 14:37:21 +0700 Subject: [PATCH 4/5] refactor(gooddata-eval): move the inherited date filter to its own PR Leave this PR with QA-30213 only (harness-error labels). The opt-in inherited date filter of QA-30214 moves to a stacked PR on top of this branch, so each one can be reviewed on its own. jira: QA-30213 risk: low Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- packages/gooddata-eval/AGENTS.md | 1 - packages/gooddata-eval/README.md | 26 -- .../src/gooddata_eval/cli/main.py | 20 - .../core/agentic/visualization.py | 30 +- .../src/gooddata_eval/core/config.py | 11 - .../core/evaluators/visualization.py | 50 +-- .../src/gooddata_eval/core/scoring.py | 93 +--- packages/gooddata-eval/tests/conftest.py | 1 - packages/gooddata-eval/tests/test_cli.py | 60 +-- .../tests/test_context_date_filter.py | 413 ------------------ 10 files changed, 13 insertions(+), 692 deletions(-) delete mode 100644 packages/gooddata-eval/tests/test_context_date_filter.py diff --git a/packages/gooddata-eval/AGENTS.md b/packages/gooddata-eval/AGENTS.md index c380c1ae9..4b33b57a2 100644 --- a/packages/gooddata-eval/AGENTS.md +++ b/packages/gooddata-eval/AGENTS.md @@ -162,7 +162,6 @@ Run this before guessing a `--model` string. | `GOODDATA_EVAL_TRACE_LABELS` | `key=value,...` labels stamped on every observation of a chat conversation via W3C baggage; `model_version` also sets the observation `version`, so set it only for a one-model run: every `--model` in one process gets the same labels. Keys must match `[A-Za-z0-9_-]+`; others are skipped with a warning. Dashboard-summary and document-takeaway runs are not labelled | | `GOODDATA_EVAL_JOIN_GENAI_TRACE` | on unless set to `0`, `false`, `no`, `off` or empty. An agentic item's chat turns ask gen-ai to open them under gd-eval's experiment root, so each item is one Langfuse trace with its scores written once. A gen-ai that drops caller baggage keeps today's separate traces; see the gotcha below | | `GD_EVAL_TIMERS` | same as `--timers` | -| `GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER` | same as `--accept-context-date-filter`; off by default. Visualization scoring accepts a date filter equal to one of the dashboard's in the item's `user_context` when the expected output has none (`check_filters`, see the README) | A gitignored `.env` at the repo root is the normal place for these; load it with `set -a && . ./.env && set +a` before the command. diff --git a/packages/gooddata-eval/README.md b/packages/gooddata-eval/README.md index 905fd5d97..a4ec369c1 100644 --- a/packages/gooddata-eval/README.md +++ b/packages/gooddata-eval/README.md @@ -133,7 +133,6 @@ gd-eval run \ | `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. | | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. | | `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail`, `dashboard_summary` and `document_takeaway`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. | -| `--accept-context-date-filter` | off | Visualization scoring also accepts a date filter the agent carried over from the dashboard in the item's `user_context` when the expected output has none. Changes verdicts, so it is opt-in; also settable via `GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER=1`. See *Date filter carried over from the dashboard* below. | | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` and `document_takeaway` items go through their own endpoints, which have no such option. | **Concurrency and workspace safety.** Agentic kinds that create workspace objects @@ -959,31 +958,6 @@ In a Langfuse dataset, put it in the item `metadata` (or in the `input` object) `null` value in `input` falls back to `metadata`. The value used must be an object or `null`; anything else fails the dataset load. -#### Date filter carried over from the dashboard (opt-in) - -A user who asks for a new chart from an open dashboard without naming a period may get -the dashboard's date filter on the chart, and the product accepts that. By default the -visualization scoring still requires the chart's date filters to equal the expected ones. -Pass `--accept-context-date-filter` to `gd-eval run`, set `GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER=1` -(`0`, `false`, `no`, `off` and empty mean off), or pass `accept_context_date_filter=True` to -`check_filters` to also accept, for the date filter only, a chart that: - -- has exactly one date filter, whose period equals a `date_filter` in - `user_context.view.dashboard.filters` (resolved against the same day), and -- belongs to an item whose expected output has no date filter. - -The dashboard's common date filter (no `using`) accepts any date dataset, one that names a -dataset accepts only that one; the one the chart used is reported in the run detail as -`filter_date_context: {"inherited": true, "dataset": ...}`, and a failure summary shows -`date : True (inherited from dashboard context)`. A chart with no date filter passes as -before; an expected date filter, a different period, ranking and attribute filters stay -strict. Applies to `visualization`, `vis_agentic` and `agentic_visualization` items; with the -variable unset the scores and the detail are exactly as without it. - -With the flag on, an expected output cannot say "all time, on purpose": an expected output -with no date filter then also accepts a chart that copied the dashboard's period. An item -that must reject the dashboard's period needs to run without the flag. - `agentic_conversation` items take the item's `user_context` as the context of the first turn. A turn in `expected_output.turns` can set its own `user_context`, which applies from that turn on until another turn changes it, as the attached context does in the UI: diff --git a/packages/gooddata-eval/src/gooddata_eval/cli/main.py b/packages/gooddata-eval/src/gooddata_eval/cli/main.py index 579ac89fd..efbbf1069 100644 --- a/packages/gooddata-eval/src/gooddata_eval/cli/main.py +++ b/packages/gooddata-eval/src/gooddata_eval/cli/main.py @@ -17,7 +17,6 @@ from gooddata_eval.cli.agentic_runner import AGENTIC_TEST_KINDS, UNGATED_AGENTIC_TEST_KINDS, run_agentic_items from gooddata_eval.core.chat.sse_client import ChatClient, set_default_item_timeout, set_default_turn_timeout from gooddata_eval.core.config import ( - ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, DEFAULT_GATE, DEFAULT_JUDGE_MODEL, JUDGE_MODEL_ENV_VAR, @@ -137,15 +136,6 @@ def _build_parser() -> argparse.ArgumentParser: "Off by default because a large run emits hundreds of lines; the same measurements " "are always in the JSON report's latency_breakdown_s. Equivalent to GD_EVAL_TIMERS=1.", ) - run.add_argument( - "--accept-context-date-filter", - dest="accept_context_date_filter", - action="store_true", - help="Visualization scoring also accepts a chart whose single date filter equals one of the " - "dashboard's in the item's user_context, when the expected output has no date filter. " - "Off by default because it changes verdicts. Equivalent to " - "GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER=1.", - ) run.add_argument( "--turn-timeout", dest="turn_timeout", @@ -319,15 +309,6 @@ def _apply_timer_flag(enabled: bool) -> None: os.environ[TIMERS_ENV_VAR] = "1" -def _apply_accept_context_date_filter_flag(enabled: bool) -> None: - """Publish --accept-context-date-filter for check_filters, like --timers. - - Only ever sets it -- never clears a value the caller exported themselves. - """ - if enabled: - os.environ[ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR] = "1" - - def _reject_power_gate_on_ungated_items(config: RunConfig, items: list) -> None: """Refuse a pass^K request the run cannot honour for every item. @@ -741,7 +722,6 @@ def sdk_factory(): def main(argv: list[str] | None = None) -> int: args = parse_args(argv if argv is not None else sys.argv[1:]) _apply_timer_flag(getattr(args, "timers", False)) - _apply_accept_context_date_filter_flag(getattr(args, "accept_context_date_filter", False)) _apply_judge_model(getattr(args, "judge_model", None)) if hasattr(args, "concurrency") and args.concurrency < 1: print("error: --concurrency must be >= 1.", file=sys.stderr) diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py index a39b3eec2..78bbb04ac 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py @@ -213,7 +213,6 @@ def _execute_single_run( max_iterations: int = _DEFAULT_MAX_ITERATIONS, requires_execution: bool = False, options: ExpectationOptions | None = None, - user_context: dict | None = None, ) -> RunResult: """Drive one full multi-turn conversation and evaluate the result.""" options = _effective_options(options, requires_execution) @@ -315,9 +314,7 @@ def _execute_single_run( if current_result.created_visualizations and current_result.created_visualizations.objects: actual_output = current_result.created_visualizations.objects[0] - eval_result, best_expected = _evaluate_against_candidates( - expected_outputs, actual_output, skill_activated, user_context=user_context - ) + eval_result, best_expected = _evaluate_against_candidates(expected_outputs, actual_output, skill_activated) eval_result = with_execution( eval_result, all_tool_call_events, current_result.text_response, options.requires_execution ) @@ -404,14 +401,7 @@ def run_agentic_visualization( try: run_results.append( _execute_single_run( - client, - conv_id_0, - question, - expected_outputs, - max_iterations, - requires_execution, - options, - user_context=user_context, + client, conv_id_0, question, expected_outputs, max_iterations, requires_execution, options ) ) finally: @@ -423,14 +413,7 @@ def run_agentic_visualization( try: run_results.append( _execute_single_run( - client, - conv_id, - question, - expected_outputs, - max_iterations, - requires_execution, - options, - user_context=user_context, + client, conv_id, question, expected_outputs, max_iterations, requires_execution, options ) ) finally: @@ -514,8 +497,6 @@ def _failed_checks(run: RunResult) -> list[dict]: ) for category in sorted(set(ev.expected_filters) | set(ev.actual_filters)) if ev.expected_filters.get(category, []) != ev.actual_filters.get(category, []) - # An inherited date filter was accepted, so it is not why the filters failed. - and not (category == "date" and ev.filter_date_inherited) ] checks.extend(differing or [FailedCheck("filters_correct")]) if not ev.sorts_correct: @@ -801,9 +782,8 @@ def _write_scores(ctx: RunTraceContext) -> None: f" expected : {sorted(ev.expected_dim_uris)}\n" f" actual : {sorted(ev.actual_dim_uris)}\n" f" Filters Correct : {ev.filters_correct}\n" - f" date : {ev.filter_date_score}" - f"{' (inherited from dashboard context)' if ev.filter_date_inherited else ''}\n" - f"{'' if ev.filter_date_inherited else _filter_diff('date', ev)}" + f" date : {ev.filter_date_score}\n" + f"{_filter_diff('date', ev)}" f" ranking : {ev.filter_ranking_score}\n" f"{_filter_diff('ranking', ev)}" f" attribute : {ev.filter_attribute_score}\n" diff --git a/packages/gooddata-eval/src/gooddata_eval/core/config.py b/packages/gooddata-eval/src/gooddata_eval/core/config.py index 6b4fce312..bb5f838ae 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/config.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/config.py @@ -17,17 +17,6 @@ def env_flag(name: str) -> bool: return os.environ.get(name, "").strip().lower() not in _ENV_FALSE -# Opt-in: accept a date filter the agent carried over from the dashboard open in -# the item's `user_context`, when the expected output states no date filter. Off by default: -# a caller that does not ask for it gets the strict date comparison. -ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR = "GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER" - - -def accept_context_date_filter() -> bool: - """Whether `check_filters` may accept a date filter inherited from the dashboard context.""" - return env_flag(ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR) - - # The judge model, overridable for comparison experiments. The default is deliberate on # two counts: gpt-4o honours temperature=0, so a given response always gets the same # verdict; and it is NOT the family under test, so the judge is not grading its own diff --git a/packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py b/packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py index 59880c6dd..10806d047 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py @@ -83,15 +83,6 @@ class EvaluationResult: format_correct: bool | None = None format_problems: list[str] = field(default_factory=list) part_present: bool | None = None - # Set only when `filter_date_score` holds because the actual date filter is the one the - # dashboard in the item's `user_context` applies (opt-in, see `check_filters`): the date - # dataset that filter used. Reported in the detail only when set. - filter_date_inherited_dataset: str | None = None - - @property - def filter_date_inherited(self) -> bool: - """Whether `filter_date_score` holds only through the dashboard-context rule.""" - return self.filter_date_inherited_dataset is not None @property def scored_without_chart(self) -> bool: @@ -462,9 +453,6 @@ def _evaluate_visualization( actual: CreatedVisualization | None, skill_activated: bool = False, today: date | None = None, - *, - user_context: Mapping | None = None, - accept_context_date_filter: bool | None = None, ) -> EvaluationResult: # One anchor for the scoring and for the filters reported beside it. check_filters # already shares an anchor across its two sides, but resolving the reported filters @@ -501,13 +489,7 @@ def _evaluate_visualization( cross_ref_valid, cross_ref_errors = validate_cross_references(actual) act_metric_uris = get_metric_uri_set(actual) act_dim_uris = get_dimension_uri_set(actual) - filter_scores = check_filters( - expected, - actual, - today, - user_context=user_context, - accept_context_date_filter=accept_context_date_filter, - ) + filter_scores = check_filters(expected, actual, today) metric_layout_problems = check_metric_layout(expected, actual) return EvaluationResult( visualization_created=True, @@ -534,7 +516,6 @@ def _evaluate_visualization( expected_buckets=bucket_layout(expected) if expected.buckets_strict else None, actual_buckets=bucket_layout(actual) if expected.buckets_strict else None, metric_layout_problems=metric_layout_problems, - filter_date_inherited_dataset=filter_scores.date_inherited_dataset, ) @@ -570,29 +551,13 @@ def _evaluate_against_candidates( actual: CreatedVisualization | None, skill_activated: bool = False, today: date | None = None, - *, - user_context: Mapping | None = None, - accept_context_date_filter: bool | None = None, ) -> tuple[EvaluationResult, CreatedVisualization | None]: if not expected_outputs: return _no_candidate_result(actual, skill_activated), None # Anchored once here too: the candidates are ranked against each other, so letting # each resolve its own today could rank them on spans from different days. today = today or date.today() - pairs = [ - ( - _evaluate_visualization( - exp, - actual, - skill_activated, - today, - user_context=user_context, - accept_context_date_filter=accept_context_date_filter, - ), - exp, - ) - for exp in expected_outputs - ] + pairs = [(_evaluate_visualization(exp, actual, skill_activated, today), exp) for exp in expected_outputs] best_result, best_expected = max(pairs, key=lambda p: (p[0].strict_pass, p[0].strict_checks_passed_count)) return best_result, best_expected @@ -659,13 +624,6 @@ def evaluation_result_detail(ev: EvaluationResult) -> dict: "actual_dim_uris": sorted(ev.actual_dim_uris), "expected_filters": ev.expected_filters, "actual_filters": ev.actual_filters, - # Nested and only when it applies: `quality_score` counts every top-level boolean, and - # a run scored without the opt-in rule must not carry the key at all. - **( - {"filter_date_context": {"inherited": True, "dataset": ev.filter_date_inherited_dataset}} - if ev.filter_date_inherited - else {} - ), "expected_sorts": ev.expected_sorts, "actual_sorts": ev.actual_sorts, **({"metric_layout_problems": ev.metric_layout_problems} if ev.metric_layout_problems else {}), @@ -700,9 +658,7 @@ def evaluate(self, item: DatasetItem, chat_result: ChatResult) -> ItemEvaluation candidates = _parse_expected(item.expected_output) actual = _extract_actual(chat_result) skill_activated = _check_visualization_skill_activated(chat_result.tool_call_events) - ev, _best_expected = _evaluate_against_candidates( - candidates, actual, skill_activated, user_context=item.user_context - ) + ev, _best_expected = _evaluate_against_candidates(candidates, actual, skill_activated) ev = with_execution(ev, chat_result.tool_call_events, chat_result.text_response, options.requires_execution) ev = with_options(ev, options, actual, chat_result.tool_call_events, render_answer_text(chat_result)) return ItemEvaluation( diff --git a/packages/gooddata-eval/src/gooddata_eval/core/scoring.py b/packages/gooddata-eval/src/gooddata_eval/core/scoring.py index 912fb837f..07e6c389e 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/scoring.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/scoring.py @@ -7,7 +7,6 @@ from dataclasses import dataclass from datetime import date, timedelta -from gooddata_eval.core.config import accept_context_date_filter as _accept_context_date_filter_from_env from gooddata_eval.core.granularity import canonical_date_uri from gooddata_eval.core.models import AacBucketRef, AacQueryField, CreatedVisualization @@ -57,16 +56,6 @@ class FilterScores: date_ok: bool ranking_ok: bool attribute_ok: bool - # Set only when `date_ok` holds because the actual date filter is the one the dashboard - # in the item's `user_context` applies (see `check_filters`), not because it matched the - # expected output: the date dataset (`using`) of that filter. The dashboard filter may name - # none, so any date dataset is accepted and the one the agent picked is recorded here. - date_inherited_dataset: str | None = None - - @property - def date_inherited(self) -> bool: - """Whether `date_ok` holds only through the dashboard-context rule.""" - return self.date_inherited_dataset is not None @property def all_ok(self) -> bool: @@ -514,96 +503,18 @@ def normalized_filters(viz: CreatedVisualization, today: date | None = None) -> return {"date": sorted(date_set), "ranking": sorted(ranking_set), "attribute": sorted(attr_set)} -def dashboard_date_filters(user_context: object) -> list[dict]: - """The date filters of the dashboard open in ``user_context`` (``view.dashboard.filters``). - - Anything that is not shaped like that -- no context, no dashboard, a non-list ``filters``, - an entry that is not a ``date_filter`` object -- contributes nothing rather than raising: - the context is the chat API's schema and is not validated by the dataset loader. - """ - if not isinstance(user_context, Mapping): - return [] - view = user_context.get("view") - dashboard = view.get("dashboard") if isinstance(view, Mapping) else None - filters = dashboard.get("filters") if isinstance(dashboard, Mapping) else None - if not isinstance(filters, list): - return [] - return [f for f in filters if isinstance(f, dict) and f.get("type") == "date_filter"] - - -def _inherited_date_filter_dataset( - exp_date: set[str], - act_date: set[str], - actual: CreatedVisualization, - dashboard_filters: list[dict], - today: date, -) -> str | None: - """The dataset of the actual date filter when it is the dashboard's own, else None. - - Holds only when the expected output states no date filter and the actual chart has - exactly one, whose span equals that of one of the dashboard's date filters resolved - against the same ``today``. The dashboard's common date filter names no dataset (it - resolves per widget), so it is compared on the actual filter's dataset and accepts any - date dataset. A dashboard date filter that names one in ``using`` accepts only that one. - """ - if exp_date or not dashboard_filters: - return None - actual_filters = [f for f in actual.query.filter_by.values() if f.get("type") == "date_filter"] - if len(actual_filters) != 1: - return None - dataset = actual_filters[0].get("using") - if not isinstance(dataset, str) or not dataset: - # A date filter that names no dataset says nothing about where it came from. - return None - fields = actual.query.fields - for dashboard_filter in dashboard_filters: - candidate = _normalize_date_filter( - {**dashboard_filter, "using": dashboard_filter.get("using") or dataset}, fields, today - ) - if json.dumps(candidate, sort_keys=True) in act_date: - return dataset - return None - - def check_filters( - expected: CreatedVisualization, - actual: CreatedVisualization, - today: date | None = None, - *, - user_context: Mapping | None = None, - accept_context_date_filter: bool | None = None, + expected: CreatedVisualization, actual: CreatedVisualization, today: date | None = None ) -> FilterScores: - """Compare the filters of two visualizations, per category. - - Every category compares exactly. One opt-in exception, for the date category only: - with ``accept_context_date_filter`` (None reads ``GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER``, - off by default), an actual date filter equal to a date filter of the dashboard in - ``user_context`` is accepted when the expected output has none -- the agent carried the - open dashboard's period into the new chart, which the product accepts. The result then - has ``date_inherited`` set. Without the flag, or without a dashboard date filter in - ``user_context``, scoring is exactly the strict comparison. - """ # One anchor for both sides: resolving each against its own date.today() would # score inconsistently for a run that straddles midnight. today = today or date.today() exp_date, exp_rank, exp_attr = _split_and_normalize_filters(expected, today) act_date, act_rank, act_attr = _split_and_normalize_filters(actual, today) - date_ok = act_date == exp_date - inherited_dataset: str | None = None - if not date_ok and user_context is not None: - enabled = ( - _accept_context_date_filter_from_env() if accept_context_date_filter is None else accept_context_date_filter - ) - if enabled: - inherited_dataset = _inherited_date_filter_dataset( - exp_date, act_date, actual, dashboard_date_filters(user_context), today - ) - date_ok = inherited_dataset is not None return FilterScores( - date_ok=date_ok, + date_ok=act_date == exp_date, ranking_ok=act_rank == exp_rank, attribute_ok=act_attr == exp_attr, - date_inherited_dataset=inherited_dataset, ) diff --git a/packages/gooddata-eval/tests/conftest.py b/packages/gooddata-eval/tests/conftest.py index f40645208..0f77eaa8a 100644 --- a/packages/gooddata-eval/tests/conftest.py +++ b/packages/gooddata-eval/tests/conftest.py @@ -26,7 +26,6 @@ def fixtures_dir() -> Path: "GOODDATA_EVAL_CHAT_TURN_TIMEOUT_S", "GOODDATA_EVAL_CHAT_ITEM_TIMEOUT_S", "GOODDATA_EVAL_JOIN_GENAI_TRACE", - "GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER", ) diff --git a/packages/gooddata-eval/tests/test_cli.py b/packages/gooddata-eval/tests/test_cli.py index 62af25bd4..ffc7404a8 100644 --- a/packages/gooddata-eval/tests/test_cli.py +++ b/packages/gooddata-eval/tests/test_cli.py @@ -1,7 +1,7 @@ # (C) 2026 GoodData Corporation import io from concurrent.futures import ThreadPoolExecutor, as_completed -from datetime import UTC, date, datetime +from datetime import UTC, datetime from unittest.mock import MagicMock import httpx @@ -9,20 +9,13 @@ import pytest from gooddata_eval.cli import main as cli_main from gooddata_eval.cli.main import _parse_model_arg -from gooddata_eval.core.config import ( - ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, - JUDGE_MODEL_ENV_VAR, - RunConfig, - accept_context_date_filter, - judge_model, -) +from gooddata_eval.core.config import JUDGE_MODEL_ENV_VAR, RunConfig, judge_model from gooddata_eval.core.connection import ( ConnectionError_, # noqa: F401 - used in test_cli_operational_error_exits_nonzero ) from gooddata_eval.core.dataset.langfuse_source import _item_from_raw -from gooddata_eval.core.models import CreatedVisualization, DatasetItem +from gooddata_eval.core.models import DatasetItem from gooddata_eval.core.runner import EvalReport, ItemReport -from gooddata_eval.core.scoring import FilterScores, check_filters from gooddata_eval.core.timing import TIMERS_ENV_VAR, timers_enabled from gooddata_eval.core.workspace import ActiveLlmProvider, ResolvedModel from rich.console import Console @@ -1204,50 +1197,3 @@ def test_routing_backend_closes_every_client() -> None: for client in (chat, summary, takeaway): client.close.assert_called_once_with() - - -def test_accept_context_date_filter_flag_is_off_by_default_and_publishes_the_env_var( - monkeypatch: pytest.MonkeyPatch, -) -> None: - base = ["run", "--host", "h", "--workspace", "w", "--dataset", "d"] - assert cli_main.parse_args(base).accept_context_date_filter is False - assert cli_main.parse_args([*base, "--accept-context-date-filter"]).accept_context_date_filter is True - - # Set, not deleted, so monkeypatch restores what the flag writes. - monkeypatch.setenv(ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, "0") - cli_main._apply_accept_context_date_filter_flag(False) - assert accept_context_date_filter() is False - cli_main._apply_accept_context_date_filter_flag(True) - assert accept_context_date_filter() is True - - -def test_accept_context_date_filter_flag_never_clears_an_exported_env_var(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv(ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, "1") - cli_main._apply_accept_context_date_filter_flag(False) - assert accept_context_date_filter() is True - - -def test_accept_context_date_filter_flag_reaches_check_filters(monkeypatch: pytest.MonkeyPatch) -> None: - def viz(filter_by: dict) -> CreatedVisualization: - return CreatedVisualization.model_validate( - { - "id": "v", - "type": "headline_chart", - "query": {"fields": {"m": {"using": "metric/m"}}, "filter_by": filter_by}, - "metrics": ["m"], - } - ) - - inherited = {"type": "date_filter", "using": "dataset/d", "granularity": "MONTH", "from": -11, "to": 0} - user_context = { - "view": {"dashboard": {"filters": [{"type": "date_filter", "granularity": "MONTH", "from": -11, "to": 0}]}} - } - - def score() -> FilterScores: - return check_filters(viz({}), viz({"f": inherited}), date(2026, 11, 12), user_context=user_context) - - monkeypatch.setenv(ACCEPT_CONTEXT_DATE_FILTER_ENV_VAR, "0") - assert score().date_ok is False # default: strict date comparison - cli_main._apply_accept_context_date_filter_flag(True) - assert score().date_ok is True - assert score().date_inherited is True diff --git a/packages/gooddata-eval/tests/test_context_date_filter.py b/packages/gooddata-eval/tests/test_context_date_filter.py deleted file mode 100644 index f66f1004f..000000000 --- a/packages/gooddata-eval/tests/test_context_date_filter.py +++ /dev/null @@ -1,413 +0,0 @@ -# (C) 2026 GoodData Corporation -"""A date filter inherited from the dashboard in the item's `user_context`. - -The rule is opt-in (`GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER`, or `accept_context_date_filter=`). -With it off, every case here must score exactly as the strict comparison does. -""" - -import json -from datetime import date -from unittest.mock import MagicMock, patch - -import pytest -from gooddata_eval.core.agentic import visualization as agentic_visualization -from gooddata_eval.core.agentic.visualization import ( - VisualizationAssertionError, - evaluate_agentic_visualization, - run_agentic_visualization, -) -from gooddata_eval.core.evaluators import get_evaluator -from gooddata_eval.core.evaluators.visualization import _evaluate_against_candidates, evaluation_result_detail -from gooddata_eval.core.models import ChatResult, CreatedVisualization, DatasetItem -from gooddata_eval.core.scoring import check_filters, dashboard_date_filters - -_ENV = "GD_EVAL_ACCEPT_CONTEXT_DATE_FILTER" -_TODAY = date(2026, 11, 12) - -# The real item from the ticket: globalmart_visualization_charts_ctx_2, run 40229016049. -_USER_CONTEXT = { - "view": { - "dashboard": { - "id": "dashboard_033", - "title": "Order Lines & Margin", - "filters": [{"type": "date_filter", "granularity": "MONTH", "from": -11, "to": 0}], - } - } -} -_INHERITED = { - "type": "date_filter", - "using": "dataset/transaction_date", - "granularity": "MONTH", - "from": -11, - "to": 0, -} - - -def _headline(filter_by: dict | None = None, metric: str = "metric/gross_margin_percent") -> dict: - return { - "id": "v", - "type": "headline_chart", - "query": {"fields": {"m": {"using": metric}}, "filter_by": filter_by or {}}, - "metrics": ["m"], - } - - -def _viz(filter_by: dict | None = None) -> CreatedVisualization: - return CreatedVisualization.model_validate(_headline(filter_by)) - - -def _date(frm: int, to: int, granularity: str = "MONTH", using: str = "dataset/transaction_date") -> dict: - return {"type": "date_filter", "using": using, "granularity": granularity, "from": frm, "to": to} - - -# ── the five acceptance-criteria cases (flag on) ───────────────────────────── - - -def test_inherited_dashboard_range_passes_and_is_marked_inherited() -> None: - scores = check_filters( - _viz(), - _viz({"f_transaction_date": _INHERITED}), - _TODAY, - user_context=_USER_CONTEXT, - accept_context_date_filter=True, - ) - assert scores.date_ok is True - assert scores.date_inherited is True - assert scores.date_inherited_dataset == "dataset/transaction_date" - assert scores.all_ok is True - - -def test_no_actual_date_filter_still_passes_without_being_inherited() -> None: - scores = check_filters(_viz(), _viz(), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True) - assert scores.date_ok is True - assert scores.date_inherited is False - - -def test_a_range_other_than_the_dashboards_fails() -> None: - scores = check_filters( - _viz(), _viz({"f": _date(-5, 0)}), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True - ) - assert scores.date_ok is False - assert scores.date_inherited is False - - -@pytest.mark.parametrize( - ("expected_filter", "actual_filter", "date_ok"), - [ - # The question names a period: the dashboard's range is not good enough... - (_date(-1, -1, "YEAR"), _INHERITED, False), - # ...and the expected one still matches exactly, as before. - (_date(-1, -1, "YEAR"), _date(-1, -1, "YEAR"), True), - # Even an expected filter equal to the dashboard's is a strict match, not "inherited". - (_INHERITED, _INHERITED, True), - ], -) -def test_an_expected_date_filter_keeps_the_strict_comparison( - expected_filter: dict, actual_filter: dict, date_ok: bool -) -> None: - expected, actual = _viz({"f": expected_filter}), _viz({"f": actual_filter}) - scores = check_filters(expected, actual, _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True) - assert scores.date_ok is date_ok - assert scores.date_inherited is False - assert scores == check_filters(expected, actual, _TODAY) - - -@pytest.mark.parametrize("user_context", [None, {}, {"view": {}}, {"view": {"dashboard": {"id": "d"}}}]) -def test_without_a_dashboard_date_filter_in_context_the_comparison_is_strict(user_context: dict | None) -> None: - scores = check_filters( - _viz(), _viz({"f": _INHERITED}), _TODAY, user_context=user_context, accept_context_date_filter=True - ) - assert scores.date_ok is False - assert scores.date_inherited is False - - -# ── further edges of the rule ──────────────────────────────────────────────── - - -def test_any_date_dataset_is_accepted_and_recorded() -> None: - scores = check_filters( - _viz(), - _viz({"f": _date(-11, 0, using="dataset/order_date")}), - _TODAY, - user_context=_USER_CONTEXT, - accept_context_date_filter=True, - ) - assert scores.date_ok is True - assert scores.date_inherited_dataset == "dataset/order_date" - - -def test_a_date_filter_that_names_no_dataset_is_never_inherited() -> None: - nameless = {k: v for k, v in _date(-11, 0).items() if k != "using"} - scores = check_filters( - _viz(), _viz({"f": nameless}), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True - ) - assert scores.date_ok is False - assert scores.date_inherited is False - - -@pytest.mark.parametrize(("actual_dataset", "date_ok"), [("dataset/order_date", True), ("dataset/ship_date", False)]) -def test_a_dashboard_filter_naming_a_dataset_accepts_only_that_one(actual_dataset: str, date_ok: bool) -> None: - context = { - "view": { - "dashboard": { - "filters": [ - {"type": "date_filter", "using": "dataset/order_date", "granularity": "MONTH", "from": -11, "to": 0} - ] - } - } - } - scores = check_filters( - _viz(), - _viz({"f": _date(-11, 0, using=actual_dataset)}), - _TODAY, - user_context=context, - accept_context_date_filter=True, - ) - assert scores.date_ok is date_ok - assert scores.date_inherited is date_ok - - -def test_the_same_span_written_absolutely_is_accepted() -> None: - # MONTH -11..0 on 2026-11-12 is 2025-12-01..2026-11-30. - absolute = {"type": "date_filter", "using": "dataset/transaction_date", "from": "2025-12-01", "to": "2026-11-30"} - scores = check_filters( - _viz(), _viz({"f": absolute}), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True - ) - assert scores.date_ok is True - assert scores.date_inherited is True - - -def test_two_actual_date_filters_fail_even_if_one_is_the_dashboards() -> None: - actual = _viz({"a": _INHERITED, "b": _date(-1, -1, "YEAR", using="dataset/order_date")}) - scores = check_filters(_viz(), actual, _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=True) - assert scores.date_ok is False - - -def test_any_of_several_dashboard_date_filters_matches() -> None: - context = { - "view": { - "dashboard": { - "filters": [ - {"type": "attribute_filter", "using": "label/region"}, - {"type": "date_filter", "granularity": "YEAR", "from": 0, "to": 0}, - {"type": "date_filter", "granularity": "MONTH", "from": -11, "to": 0}, - ] - } - } - } - assert len(dashboard_date_filters(context)) == 2 - scores = check_filters( - _viz(), _viz({"f": _INHERITED}), _TODAY, user_context=context, accept_context_date_filter=True - ) - assert scores.date_ok is True - - -def test_ranking_and_attribute_filters_are_not_relaxed() -> None: - attribute = {"type": "attribute_filter", "using": "label/region", "state": {"include": ["EMEA"]}} - scores = check_filters( - _viz(), - _viz({"f": _INHERITED, "r": attribute}), - _TODAY, - user_context=_USER_CONTEXT, - accept_context_date_filter=True, - ) - assert scores.date_ok is True - assert scores.attribute_ok is False - assert scores.all_ok is False - - -@pytest.mark.parametrize("malformed", ["text", {"view": "x"}, {"view": {"dashboard": {"filters": "x"}}}]) -def test_a_malformed_context_is_ignored_not_raised(malformed: object) -> None: - assert dashboard_date_filters(malformed) == [] - - -# ── the flag ──────────────────────────────────────────────────────────────── - - -@pytest.mark.parametrize("value", [None, "", "0", "false", "no", "off"]) -def test_flag_off_scores_strictly(monkeypatch: pytest.MonkeyPatch, value: str | None) -> None: - if value is not None: - monkeypatch.setenv(_ENV, value) - expected, actual = _viz(), _viz({"f": _INHERITED}) - scores = check_filters(expected, actual, _TODAY, user_context=_USER_CONTEXT) - assert scores.date_ok is False - assert scores == check_filters(expected, actual, _TODAY) - - -@pytest.mark.parametrize("value", ["1", "true", "yes", "on"]) -def test_flag_on_from_the_environment(monkeypatch: pytest.MonkeyPatch, value: str | None) -> None: - monkeypatch.setenv(_ENV, value) - scores = check_filters(_viz(), _viz({"f": _INHERITED}), _TODAY, user_context=_USER_CONTEXT) - assert scores.date_ok is True - assert scores.date_inherited is True - - -def test_the_keyword_overrides_the_environment(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv(_ENV, "1") - scores = check_filters( - _viz(), _viz({"f": _INHERITED}), _TODAY, user_context=_USER_CONTEXT, accept_context_date_filter=False - ) - assert scores.date_ok is False - - -@pytest.mark.parametrize( - "actual_filter_by", - [{}, {"f": _INHERITED}, {"f": _date(-5, 0)}, {"a": _INHERITED, "b": _date(0, 0, "YEAR")}], -) -@pytest.mark.parametrize("expected_filter_by", [{}, {"f": _date(-1, -1, "YEAR")}]) -def test_flag_off_detail_is_byte_identical_with_and_without_user_context( - expected_filter_by: dict, actual_filter_by: dict -) -> None: - expected, actual = [_viz(expected_filter_by)], _viz(actual_filter_by) - before, _ = _evaluate_against_candidates(expected, actual, True, _TODAY) - after, _ = _evaluate_against_candidates(expected, actual, True, _TODAY, user_context=_USER_CONTEXT) - assert after == before - dump = json.dumps(evaluation_result_detail(before), sort_keys=True, default=sorted) - assert json.dumps(evaluation_result_detail(after), sort_keys=True, default=sorted) == dump - assert "filter_date_context" not in dump - - -# ── the real payload from the ticket, through both evaluation paths ────────── - - -def _ticket_item() -> DatasetItem: - return DatasetItem( - id="globalmart_visualization_charts_ctx_2", - dataset_name="globalmart_visualization_charts_ctx", - test_kind="visualization", - question="Show Gross Margin Percent as a headline.", - expected_output={"visualization": _headline()}, - user_context=_USER_CONTEXT, - ) - - -def _ticket_chat(metric: str = "metric/gross_margin_percent") -> ChatResult: - return ChatResult.model_validate( - { - "createdVisualizations": { - "objects": [_headline({"f_transaction_date": _INHERITED}, metric=metric)], - "reasoning": "", - }, - "toolCallEvents": [], - } - ) - - -def test_ticket_item_single_shot_fails_with_the_flag_off() -> None: - result = get_evaluator("visualization").evaluate(_ticket_item(), _ticket_chat()) - assert result.passed is False - assert result.detail["filter_date_score"] is False - assert "filter_date_context" not in result.detail - - -def test_ticket_item_single_shot_passes_with_the_flag_on(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv(_ENV, "1") - result = get_evaluator("visualization").evaluate(_ticket_item(), _ticket_chat()) - assert result.passed is True - assert result.detail["filter_date_score"] is True - assert result.detail["filter_date_context"] == {"inherited": True, "dataset": "dataset/transaction_date"} - - -def _agentic_client(chat: ChatResult) -> MagicMock: - client = MagicMock() - client.create_conversation.return_value = "conv-1" - client.send_message.return_value = chat - return client - - -@pytest.mark.parametrize(("flag", "passes"), [(None, False), ("1", True)]) -def test_ticket_item_agentic_run(monkeypatch: pytest.MonkeyPatch, flag: str | None, passes: bool) -> None: - if flag is not None: - monkeypatch.setenv(_ENV, flag) - with patch.object(agentic_visualization, "ChatClient", return_value=_agentic_client(_ticket_chat())): - summary = run_agentic_visualization( - host="https://example.com", - token="tok", - workspace_id="ws", - question="Show Gross Margin Percent as a headline.", - expected_outputs=[_viz()], - k=1, - user_context=_USER_CONTEXT, - ) - ev = summary.best.eval_result - assert ev.strict_pass is passes - assert ev.filter_date_score is passes - assert ev.filter_date_inherited is passes - - -def test_strict_check_summary_shows_the_inherited_date(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv(_ENV, "1") - # A wrong metric keeps the item failing, so the summary is printed. - chat = _ticket_chat(metric="metric/revenue") - with ( - patch.object(agentic_visualization, "ChatClient", return_value=_agentic_client(chat)), - pytest.raises(VisualizationAssertionError) as exc_info, - ): - evaluate_agentic_visualization( - host="https://example.com", - token="tok", - workspace_id="ws", - question="Show Gross Margin Percent as a headline.", - expected_outputs=[_viz()], - k=1, - user_context=_USER_CONTEXT, - ) - message = str(exc_info.value) - assert " date : True (inherited from dashboard context)\n" in message - assert "Metrics Correct : False" in message - - -def test_strict_check_summary_is_unchanged_with_the_flag_off() -> None: - with ( - patch.object(agentic_visualization, "ChatClient", return_value=_agentic_client(_ticket_chat())), - pytest.raises(VisualizationAssertionError) as exc_info, - ): - evaluate_agentic_visualization( - host="https://example.com", - token="tok", - workspace_id="ws", - question="Show Gross Margin Percent as a headline.", - expected_outputs=[_viz()], - k=1, - user_context=_USER_CONTEXT, - ) - message = str(exc_info.value) - assert " date : False\n" in message - assert "inherited" not in message - - -def test_failed_checks_leave_out_an_accepted_inherited_date(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv(_ENV, "1") - # An extra attribute filter fails the filters; the inherited date filter must not be blamed. - chat = ChatResult.model_validate( - { - "createdVisualizations": { - "objects": [ - _headline( - { - "f_transaction_date": _INHERITED, - "a": {"type": "attribute_filter", "using": "label/region", "state": {"include": ["x"]}}, - } - ) - ], - "reasoning": "", - }, - "toolCallEvents": [], - } - ) - with patch.object(agentic_visualization, "ChatClient", return_value=_agentic_client(chat)): - summary = run_agentic_visualization( - host="https://example.com", - token="tok", - workspace_id="ws", - question="Show Gross Margin Percent as a headline.", - expected_outputs=[_viz()], - k=1, - user_context=_USER_CONTEXT, - ) - run = summary.best - assert run.eval_result.filter_date_inherited is True - assert run.eval_result.filters_correct is False - labels = [c.get("label") for c in agentic_visualization._failed_checks(run) if c.get("name") == "filters_correct"] - assert "date" not in labels - assert "attribute" in labels From 14e1edd914452a9a6c3c3442b07826a54a508ed9 Mon Sep 17 00:00:00 2001 From: "my.nguyen" <my.nguyen@gooddata.com> Date: Tue, 17 Nov 2026 14:58:57 +0700 Subject: [PATCH 5/5] docs(gooddata-eval): drop the README additions from this PR jira: QA-30213 risk: low Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- packages/gooddata-eval/README.md | 56 -------------------------------- 1 file changed, 56 deletions(-) diff --git a/packages/gooddata-eval/README.md b/packages/gooddata-eval/README.md index a4ec369c1..dfda520f4 100644 --- a/packages/gooddata-eval/README.md +++ b/packages/gooddata-eval/README.md @@ -265,62 +265,6 @@ the same reason. When some runs were ungraded the note says so instead of callin `agentic_conversation` has no K gate, so `--gate power` is refused for a dataset containing one rather than labelling a report `power` that only part of the dataset was decided under. -**Which runs failed, and why.** A failure message names every failing run right after the gate note, with the -reason its loop stopped and only the checks it failed (expected/actual where the kind knows them), then the best -run's summary as before: - -``` -Gate pass^3 failed: 2/3 runs passed — unstable, not a clean failure. Failing runs: run 2/3 FAIL (loop exit: success): filters_correct[date: expected none, actual ...]. -``` - -The same data is in each `failed_runs[]` record: `exit_reason` (the run's `LoopExit`, `null` for a kind without -one), `outcome` (see below) and, for the visualization, alert, metric and dashboard skills, `failed_checks` -(`[{"name", "label"?, "expected"?, "actual"?}]`, each `expected`/`actual` longer than 160 characters stored cut to -that length; the run's `detail` keeps the whole value). - -None of this changes a verdict, a score or an exception type; it only says more about each failure. - -### Harness errors are labelled, not rescored - -A run can fail because the test harness failed, not the agent: the simulated user's own model call fails, or the -judge cannot grade the answer. Each failing run is labelled in `failed_runs[].outcome` -(`{"status": "fail" | "error", "reason": ...}`) by one rule (`core/outcome.py`, `classify_run`). The rule is -fail-closed -- only these causes are `error`, anything else, an unknown cause included, is `fail`: - -| Cause | Label | -|---|---| -| `simulated_user_failed` | `error` | -| the judge or scorer could not grade the run (`judge_failed`) | `error` | -| `chat_error` (a gen-ai error event, a broken stream, a 5xx after its retries) | `fail` -- on the product path, a customer sees it too | -| `agent_timeout`, `turn_incomplete` (the agent was too slow, or ended its turn without an answer) | `fail` | -| `success`, `budget_exhausted`, `agent_silent`, `not_run`, anything else | `fail` | - -The label decides nothing: an `error` run counts as a run that did not pass, so every gate, pass rate and score is -what it was before labels existed. It shows up in four places: - -- the "Failing runs" part of the message: `run 1/3 ERROR (simulated_user_failed)`, without the run's checks, which - failed because the harness did; -- `failed_runs[].outcome`; -- on the raised exception, `outcome` = `{"status": "error", "reason": ...}` when **every** run of the item errored - (the item says nothing about the agent; its message starts `Harness error, no run was scored (<reason>).`), and - `shadow` -- what a candidate rule that leaves `error` runs out of the gate (at least ceil(K/2) scored runs, else - `error`) would have given: `{"verdict": "pass" | "fail" | "error", "runs_scored", "runs_total"}`. Recorded so the - rule can be judged on real runs; nothing uses it; -- in Langfuse, an `error` run gets an extra score `eval_error` = 1, commented with the reason. Its other scores are - written as always. - -A runner turns these into its own report. The gdc-nas Tavern harness, for example, adds them to the JUnit testcase -as properties and keeps the failure a failure. - -Applies to the agentic visualization, alert, metric, dashboard, KDA, anomaly detection and what-if kinds, and to -judge failures of the general question, guardrail and document skill kinds. `agentic_conversation` records its -turns' exits but does not label them yet. - -**Changed exit value:** a run whose turn timed out or ended without a final answer used to record `exit_reason` -`chat_error`; it now records `agent_timeout` or `turn_incomplete`, in the run detail and in `failed_runs[]`. Filters -on `exit_reason == "chat_error"` see fewer runs. The item budget (`--item-timeout`) runs from conversation creation, -so an `agent_timeout` from it can include the simulated user's replies and the chat retries' backoff. - Each item additionally carries a per-phase breakdown: ```json