Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions packages/gooddata-eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -752,6 +752,9 @@ Item-level keys, all optional except the candidates:
`{"metric/net_sales": {"divide_by": 1000000, "decimals": 1, "suffix": "M"}}`.
- `requires_visualization_part` — the chart must come back as a part of the answer, not only as
the arguments of the chart tool call.
- `date_filter_from_context` — `true` or `{"dataset": "<date dataset>"}`: a candidate without a
date filter also accepts the chart carrying a date filter of the dashboard in the item's
`user_context`. See *Date filter carried over from the dashboard* below.

Switches on a candidate chart:

Expand Down Expand Up @@ -902,6 +905,44 @@ In a Langfuse dataset, put it in the item `metadata` (or in the `input` object)
`null` value in `input` falls back to `metadata`. The value used must be an object or `null`;
anything else fails the dataset load.

#### Date filter carried over from the dashboard

A user who asks for a new chart from an open dashboard without naming a period may get the
dashboard's date filter on the chart, and the product accepts that as well as a chart with no
date filter. An item says so with `date_filter_from_context`:

```json
"expected_output": {
"visualization": {"type": "headline_chart", "query": {"fields": {"m": {"using": "metric/gross_margin_pct"}}}, "metrics": ["m"]},
"date_filter_from_context": {"dataset": "dataset/transaction_date"}
}
```

Each candidate without a date filter then also counts once more per date filter in
`user_context.view.dashboard.filters`, with that filter added. Both sides resolve against the
same day, so a relative and an absolute period that cover the same days match. The date dataset
of the added filter is, in order: the `dataset` the key states; the one the dashboard filter
names in `using` (a bare id such as `order_date` reads as `dataset/order_date`); for the
dashboard's common date filter, which names none, the one the chart's only date filter uses.
`true` therefore accepts any date dataset for the common date filter. The key's `dataset` must
be a bare id or a `dataset/` URI; anything else, such as `label/region` or a misspelled
`datset/order_date`, is an error. A dashboard filter whose `using` is not a date dataset adds
no candidate.

A candidate that states a date filter is not extended: a question that names a period is scored
on that period only. A chart with two date filters matches no added candidate. Ranking and
attribute filters are unaffected.

When the chart matches an added candidate, the run detail carries
`filter_date_context: {"inherited": true, "dataset": ...}` and a failure summary shows
`date : True (matches the dashboard date filter in user_context)`. The expected output reported
is still the candidate the item states; `expected_filters` shows the dashboard filter it was
scored with. A stated candidate that matched the date wins over an added one it ties with. Without the key an item is
scored exactly as before, so an item that must reject the dashboard's period ("all time")
simply leaves it out. The key on an item whose `user_context` has no dashboard date filter is
an error, raised before any request. Applies to `visualization`, `vis_agentic` and
`agentic_visualization` items; `agentic_conversation` turns do not read it.

`agentic_conversation` items take the item's `user_context` as the context of the first
turn. A turn in `expected_output.turns` can set its own `user_context`, which applies
from that turn on until another turn changes it, as the attached context does in the UI:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
ExpectationOptions,
_check_visualization_skill_activated,
_evaluate_against_candidates,
check_context_date_filter,
evaluation_result_detail,
with_execution,
with_options,
Expand Down Expand Up @@ -213,6 +214,7 @@ def _execute_single_run(
max_iterations: int = _DEFAULT_MAX_ITERATIONS,
requires_execution: bool = False,
options: ExpectationOptions | None = None,
user_context: dict | None = None,
) -> RunResult:
"""Drive one full multi-turn conversation and evaluate the result."""
options = _effective_options(options, requires_execution)
Expand Down Expand Up @@ -314,7 +316,13 @@ def _execute_single_run(
if current_result.created_visualizations and current_result.created_visualizations.objects:
actual_output = current_result.created_visualizations.objects[0]

eval_result, best_expected = _evaluate_against_candidates(expected_outputs, actual_output, skill_activated)
eval_result, best_expected = _evaluate_against_candidates(
expected_outputs,
actual_output,
skill_activated,
user_context=user_context,
context_date_filter=options.date_filter_from_context,
)
eval_result = with_execution(
eval_result, all_tool_call_events, current_result.text_response, options.requires_execution
)
Expand Down Expand Up @@ -379,13 +387,16 @@ def run_agentic_visualization(
ValueError: ``expected_outputs`` and ``options.expects_visualization`` disagree. An
empty candidate list is only right for an item that expects no chart, and the
scorer would otherwise read the default ``expects_visualization`` and fail it.
Also raised, before any request, for ``date_filter_from_context`` on an item whose
``user_context`` has no dashboard date filter (``check_context_date_filter``).
"""
expects_visualization = (options or ExpectationOptions()).expects_visualization
if expects_visualization != bool(expected_outputs):
raise ValueError(
"expected_outputs must list the candidate charts unless options.expects_visualization is false, "
f"got {len(expected_outputs)} candidate(s) with expects_visualization={expects_visualization}"
)
check_context_date_filter(options or ExpectationOptions(), user_context)
client = ChatClient(
host=host,
token=token,
Expand All @@ -401,7 +412,14 @@ def run_agentic_visualization(
try:
run_results.append(
_execute_single_run(
client, conv_id_0, question, expected_outputs, max_iterations, requires_execution, options
client,
conv_id_0,
question,
expected_outputs,
max_iterations,
requires_execution,
options,
user_context=user_context,
)
)
finally:
Expand All @@ -413,7 +431,14 @@ def run_agentic_visualization(
try:
run_results.append(
_execute_single_run(
client, conv_id, question, expected_outputs, max_iterations, requires_execution, options
client,
conv_id,
question,
expected_outputs,
max_iterations,
requires_execution,
options,
user_context=user_context,
)
)
finally:
Expand Down Expand Up @@ -789,7 +814,8 @@ def _write_scores(ctx: RunTraceContext) -> None:
f" expected : {sorted(ev.expected_dim_uris)}\n"
f" actual : {sorted(ev.actual_dim_uris)}\n"
f" Filters Correct : {ev.filters_correct}\n"
f" date : {ev.filter_date_score}\n"
f" date : {ev.filter_date_score}"
f"{' (matches the dashboard date filter in user_context)' if ev.filter_date_inherited else ''}\n"
f"{_filter_diff('date', ev)}"
f" ranking : {ev.filter_ranking_score}\n"
f"{_filter_diff('ranking', ev)}"
Expand Down
Loading
Loading