diff --git a/packages/gooddata-eval/AGENTS.md b/packages/gooddata-eval/AGENTS.md index f7c298e0d..772dd7aa3 100644 --- a/packages/gooddata-eval/AGENTS.md +++ b/packages/gooddata-eval/AGENTS.md @@ -63,7 +63,8 @@ implementation, sometimes under different `test_kind` strings (`search_tool` vs its own shape. `test_kind` on the item is what labels the result, not the evaluator class, which is why `knowledge_question` can reuse `GeneralQuestionEvaluator` verbatim. `dashboard_summary` items additionally need `summary_input`, `document_takeaway` items -`takeaway_input`, `catalog_metadata` items `catalog_input`. +`takeaway_input`, `catalog_metadata` items `catalog_input`, `agentic_agent_switcher` items +`agent_switcher_input`. ## Running the pipeline diff --git a/packages/gooddata-eval/README.md b/packages/gooddata-eval/README.md index 9749259d1..3a54cd24a 100644 --- a/packages/gooddata-eval/README.md +++ b/packages/gooddata-eval/README.md @@ -323,8 +323,8 @@ their eval kind, and states latency for each: `agentic_general_question`, `agentic_guardrail` and `agentic_search`. - **Multi-turn** — kinds that drive a simulated user or fixture turns: `agentic_visualization` (and `vis_agentic`), `agentic_metric_skill`, `agentic_alert_skill`, `agentic_dashboard_skill`, - `agentic_document_skill`, `agentic_kda_skill`, `agentic_anomaly_detection`, `agentic_what_if` and - `agentic_conversation`. + `agentic_document_skill`, `agentic_kda_skill`, `agentic_anomaly_detection`, `agentic_what_if`, + `agentic_agent_switcher` and `agentic_conversation`. ```json "summary": { @@ -912,6 +912,66 @@ A 200 answer with no usable text (a null value without a note, or the placeholde `OPENAI_API_KEY`); re-run it after changing them. Request errors are handled as for `document_takeaway`; the CI entry point is `evaluate_catalog_metadata`. +### `agentic_agent_switcher` items + +Agent-switcher items drive one conversation through several agents. The conversation is created +with `agent_id`; a turn with `switch_to` calls +`POST /api/v1/ai/workspaces/{ws}/chat/conversations/{id}/switchAgent` before its message is sent. +Each of the `--runs` replays every turn on a new conversation. Switching needs the agent-switching +feature flag; a 404 on the switch fails the run and says so. In a Langfuse dataset the block goes in +the item's `metadata` or its `input` object; an item carrying it is an `agentic_agent_switcher` +item unless it declares another `test_kind`. `question` only labels the item. + +```json +{ + "id": "switch-001", + "dataset_name": "agent_switcher", + "test_kind": "agentic_agent_switcher", + "question": "Plan with the planner agent, build the alert with the builder agent", + "agent_switcher_input": { + "agent_id": "planner", + "turns": [ + {"message": "I want to watch revenue every week and hear when it drops below 1000."}, + { + "message": "Set that up now.", + "switch_to": "builder", + "checks": { + "must_include": ["Restates the plan the user gave earlier: weekly revenue, below 1000."], + "must_not_include": ["Asks the user to repeat the plan."], + "skills_used": ["alert"], + "skills_not_used": ["visualization"], + "alert_metric": "revenue" + } + }, + {"message": "Also chart revenue by region.", "checks": {"visualization": {"metrics": ["revenue"], "attributes": ["region"]}}} + ] + } +} +``` + +`agent_id` and at least one turn are required; unknown keys are rejected at every level. Every +`checks` key is optional: + +- `must_include`, `must_not_include` — LLM judge criteria for this turn's answer. The judge sees the + transcript so far, each earlier answer labelled with the agent that gave it. +- `skills_used`, `skills_not_used` — skills whose tools this turn must (not) run: the `skillsInUse` + names on the turn's `toolCall` items, read from the conversation items after the last turn. A skill + still active from an earlier turn counts as soon as one of its tools runs, with no new `set_skills` + call; a skill that was requested but whose tools never ran does not count. When the items cannot be + read, every skills check fails. +- `alert_metric` — the primary (first) measure of the turn's alert proposal is this metric id, with or + without the `metric/` prefix; the comparison measure of a metric-to-metric alert does not count. +- `visualization` — a chart created in this turn uses these `metrics` and `attributes` (label ids), + with or without the `metric/` / `label/` prefix; extra fields are allowed. + +Two checks run on every run, from the conversation items read back after the last turn: +`agent_stamped` (every assistant item of a turn carries the agent active at that turn) and +`switch_marker` (one system marker with `trigger` `agent_switched` per switch, carrying the new +agent). A turn is a user item and the items after it up to the next user item. A switch to the +agent already bound changes nothing and expects no marker. Per-turn checks are reported as +`turn_`, e.g. `turn2_skills_used`, `turn2_include_0`. The CI entry point is +`evaluate_agentic_agent_switcher`, which takes the whole item; `--agent-id` does not apply. + ### Items asked in a dashboard context (`user_context`) An item can be asked the way a user asks from an open dashboard or an attached widget. @@ -1013,6 +1073,7 @@ from that turn on until another turn changes it, as the attached context does in | `dashboard_summary` | Dashboard summary (via `/summary` endpoint) scored against a rubric by LLM | `[llm-judge]` | | `document_takeaway` | One short takeaway for a page (via `/takeaway` endpoint): deterministic format checks plus a rubric scored by LLM | `[llm-judge]` | | `catalog_metadata` | Analytics Catalog title or description for one object (via the `analyticsCatalog/generate*` endpoints): format checks plus a rubric scored by LLM | `[llm-judge]` | +| `agentic_agent_switcher` | One conversation switched between agents: each answer stamped with the active agent, one switch marker per switch, and per-turn judge, skill, alert and chart checks | `agent_switcher_input`; `[llm-judge]` for judge criteria | ## Optional extras diff --git a/packages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.py b/packages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.py index e596c6128..55a1ad917 100644 --- a/packages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.py +++ b/packages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.py @@ -12,6 +12,7 @@ from gooddata_eval.core.agentic._gate import DEFAULT_GATE, EvalGate, normalize_gate from gooddata_eval.core.agentic._langfuse import make_langfuse_client from gooddata_eval.core.agentic._trace_linker import BackgroundTraceLinker, SubmitTraceLink, run_trace_link_inline +from gooddata_eval.core.agentic.agent_switcher import evaluate_agentic_agent_switcher from gooddata_eval.core.agentic.alert_skill import evaluate_agentic_alert_skill from gooddata_eval.core.agentic.anomaly_detection import evaluate_agentic_anomaly_detection from gooddata_eval.core.agentic.conversation import ConversationFixture, evaluate_agentic_conversation @@ -62,6 +63,7 @@ class _LfKw(TypedDict, total=False): "agentic_dashboard_summary", "agentic_anomaly_detection", "agentic_what_if", + "agentic_agent_switcher", } ) @@ -368,6 +370,17 @@ def _dispatch_agentic( user_context=item.user_context, **lf_kw, ) + elif kind == "agentic_agent_switcher": + # The item names its own agents and carries its own user_context. + return evaluate_agentic_agent_switcher( + host=host, + token=token, + workspace_id=workspace_id, + dataset_item=item, + k=k, + gate=gate, + **lf_kw, + ) elif kind == "agentic_conversation": fixture_data = eo.get("fixture") or eo if isinstance(eo, dict) else {} return evaluate_agentic_conversation( diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/__init__.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/__init__.py index b324943de..a7475042f 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/agentic/__init__.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/__init__.py @@ -1,4 +1,11 @@ # (C) 2026 GoodData Corporation. All rights reserved. +from gooddata_eval.core.agentic.agent_switcher import ( + AgenticAgentSwitcherSummary, + AgentSwitcherAssertionError, + AgentSwitcherRunResult, + evaluate_agentic_agent_switcher, + run_agentic_agent_switcher, +) from gooddata_eval.core.agentic.alert_skill import ( AgenticAlertSummary, AlertEvaluation, @@ -85,6 +92,9 @@ run_agentic_report_skill = run_agentic_document_skill __all__ = [ + "AgentSwitcherAssertionError", + "AgentSwitcherRunResult", + "AgenticAgentSwitcherSummary", "AgenticAlertSummary", "AgenticDashboardSummary", "AgenticGeneralQuestionSummary", @@ -125,6 +135,7 @@ "TurnDefinition", "TurnResult", "VisualizationAssertionError", + "evaluate_agentic_agent_switcher", "evaluate_agentic_alert_skill", "evaluate_agentic_conversation", "evaluate_agentic_dashboard_skill", @@ -136,6 +147,7 @@ "evaluate_agentic_report_skill", "evaluate_agentic_search_tool", "evaluate_agentic_visualization", + "run_agentic_agent_switcher", "run_agentic_alert_skill", "run_agentic_conversation", "run_agentic_dashboard_skill", diff --git a/packages/gooddata-eval/src/gooddata_eval/core/agentic/agent_switcher.py b/packages/gooddata-eval/src/gooddata_eval/core/agentic/agent_switcher.py new file mode 100644 index 000000000..3eb1e1054 --- /dev/null +++ b/packages/gooddata-eval/src/gooddata_eval/core/agentic/agent_switcher.py @@ -0,0 +1,764 @@ +# (C) 2026 GoodData Corporation. All rights reserved. +"""Agentic agent-switcher evaluation: one conversation, switched between agents mid-way. + +Each run replays the item's turns on a new conversation created with ``agent_id``. A turn +with ``switch_to`` calls ``switchAgent`` before its message is sent. Per-turn checks come +from the item: + +- ``must_include`` / ``must_not_include``: LLM judge criteria, graded against the + transcript so far (earlier answers labelled with the agent that gave them). +- ``skills_used`` / ``skills_not_used``: skills whose tools the turn ran, read from the + stored items: ``skillsInUse`` of the turn's ``toolCall`` items. A skill carried over from + an earlier turn counts once one of its tools runs; a requested skill whose tools never + ran does not. +- ``alert_metric``: the primary measure of the turn's alert proposal is that metric. +- ``visualization``: the turn's created chart uses these metrics and attributes. + +The skills checks and two run checks are read from the stored items after the last turn: + +- ``agent_stamped``: every assistant item of a turn carries the agent active at that turn. +- ``switch_marker``: one system marker per real switch, carrying the new agent, before the + turn it was made for. A switch to the agent already bound is a no-op and has no marker. +""" + +from __future__ import annotations + +import time +from dataclasses import dataclass, field +from typing import Any + +import httpx + +from gooddata_eval.core.agentic._failed_runs import build_failed_runs +from gooddata_eval.core.agentic._failure_message import FailedCheck, failed_check_records +from gooddata_eval.core.agentic._gate import ( + DEFAULT_GATE, + EvalGate, + gate_passed, + log_gate_scores, + stamp_gate_metadata, +) +from gooddata_eval.core.agentic._outcome import ( + agentic_detail, + agentic_success, + attach_labels, + chat_failure_exit, + failure_message_head, + label_runs, + mark_unscored_run, + raise_agentic_failure, +) +from gooddata_eval.core.agentic._trace_linker import ( + RunIdentity, + RunTraceContext, + SubmitTraceLink, + conversations_to_link, + open_item_trace, + run_trace_link_inline, + submit_trace_scoring, + utc_now, +) +from gooddata_eval.core.agentic.alert_skill import render_turn_for_simulated_user +from gooddata_eval.core.chat.sse_client import ChatClient, ChatError +from gooddata_eval.core.config import ReasoningEffort +from gooddata_eval.core.evaluators._llm_judge import JudgeResponseError, LLMJudge +from gooddata_eval.core.evaluators._rubric import Criterion, grade_rubric +from gooddata_eval.core.langfuse.item_scope import item_scope +from gooddata_eval.core.models import ( + AgenticAssertionError, + AgenticEvalOutcome, + AgentSwitcherInput, + AgentSwitcherTurn, + AgentSwitcherVisualization, + ChatResult, + DatasetItem, + LoopExit, + ReasoningStepEvent, + ToolCallEvent, + build_latency_breakdown, + shift_and_index_events, +) +from gooddata_eval.core.scoring import get_dimension_uri_set, get_metric_uri_set +from gooddata_eval.core.timing import PhaseTimings, sum_timings + +_TEST_KIND = "agentic_agent_switcher" +_DEFAULT_K = 1 +# Per earlier answer in the judge transcript; the turn under grading is never cut. +_TRANSCRIPT_ANSWER_CHARS = 2000 + +_CONTEXT_STEP = ( + "The INPUT is a conversation between a user and an analytics assistant, ending with the user's " + "LATEST MESSAGE. Each earlier answer is labelled with the agent that gave it; the conversation may " + "have been switched from one agent to another. Use the conversation only to resolve what the " + "criterion refers to (for example 'the plan the user gave earlier'). Grade the ACTUAL OUTPUT, the " + "answer to the latest message, not the earlier answers." +) + +_POSITIVE_STEPS = [ + _CONTEXT_STEP, + "Read the EXPECTED OUTPUT: one criterion the answer must satisfy.", + "Score 1 if the actual output clearly satisfies the criterion.", + "Score 0 if the criterion is missing, contradicted, or only partially addressed.", +] + +# A presence question, inverted in code -- see the same split in evaluators/takeaway.py. +_VIOLATION_STEPS = [ + _CONTEXT_STEP, + "Read the CHARACTERISTIC described in EXPECTED OUTPUT, including any exceptions it states.", + "Score 1 if the actual output clearly exhibits the described characteristic.", + "Score 0 if it does not exhibit it.", +] + + +class AgentSwitcherAssertionError(AgenticAssertionError): + """Raised when an agent-switcher evaluation fails its gate.""" + + +@dataclass +class TurnRecord: + """What one turn sent, which agent answered, and how its checks went.""" + + turn: int + message: str + agent_id: str + switch_to: str | None = None + answer: str = "" + answering_agents: list[str] = field(default_factory=list) + # Skills whose tools ran in the turn (`skillsInUse` of its toolCall items). + skills: list[str] = field(default_factory=list) + checks: dict[str, bool] = field(default_factory=dict) + reasons: dict[str, str] = field(default_factory=dict) + + def as_record(self) -> dict[str, Any]: + return { + "turn": self.turn, + "message": self.message, + "switch_to": self.switch_to, + "agent_id": self.agent_id, + "answering_agents": self.answering_agents, + "answer": self.answer, + "skills": self.skills, + "checks": self.checks, + "reasons": self.reasons, + } + + +@dataclass +class AgentSwitcherRunResult: + """One replay of the item's turns on its own conversation.""" + + conversation_id: str + turns: list[TurnRecord] = field(default_factory=list) + # Every check that ran, by name: per turn `turn_`, per run `agent_stamped`, `switch_marker`. + checks: dict[str, bool] = field(default_factory=dict) + failed: list[FailedCheck] = field(default_factory=list) + # A switch, chat or items call that failed; the run stopped there. + error: str | None = None + # Set only when every graded check passed and a judge criterion went ungraded. + judge_error: str | None = None + exit_reason: LoopExit = LoopExit.SUCCESS + reasoning_steps: list[str] = field(default_factory=list) + response_id: str | None = None + tool_call_events: list[ToolCallEvent] = field(default_factory=list) + reasoning_step_events: list[ReasoningStepEvent] = field(default_factory=list) + timings: PhaseTimings = field(default_factory=PhaseTimings) + run_latency_s: float = 0.0 + + @property + def passed(self) -> bool: + return self.error is None and self.judge_error is None and all(self.checks.values()) + + def fail(self, check: FailedCheck) -> None: + self.checks[check.name] = False + self.failed.append(check) + + +@dataclass +class AgenticAgentSwitcherSummary: + """Aggregated outcome of K runs.""" + + run_results: list[AgentSwitcherRunResult] + pass_at_k: bool + pass_power_k: bool + best: AgentSwitcherRunResult + + @property + def scored_run_results(self) -> list[AgentSwitcherRunResult]: + return [r for r in self.run_results if r.judge_error is None] + + @property + def judge_errors(self) -> list[str]: + return [r.judge_error for r in self.run_results if r.judge_error is not None] + + +@dataclass +class _Judges: + positive: LLMJudge + violation: LLMJudge + + +def _strip_type(ref: str) -> str: + """`metric/revenue` -> `revenue`; a bare id stays as it is.""" + return ref.split("/", 1)[1] if "/" in ref else ref + + +def _proposal_metric_ids(chat_result: ChatResult) -> list[str]: + """Primary measure id of each of the turn's alert proposals. + + - primary = first of `alert.execution.measures` (gen-ai puts the alerted metric first); + - a second measure is only the comparison side, so it is not read. + """ + ids: list[str] = [] + for proposal in chat_result.alert_proposals: + alert = proposal.get("alert") if isinstance(proposal, dict) else None + execution = alert.get("execution") if isinstance(alert, dict) else None + measures = execution.get("measures") if isinstance(execution, dict) else None + if not isinstance(measures, list) or not measures: + continue + node: Any = measures[0] + for key in ("definition", "measure", "item", "identifier", "id"): + node = node.get(key) if isinstance(node, dict) else None + if isinstance(node, str): + ids.append(node) + return ids + + +def _visualization_problem(expected: AgentSwitcherVisualization, chat_result: ChatResult) -> str | None: + """None when a created chart uses every expected metric and attribute, else what was created.""" + objects = chat_result.created_visualizations.objects if chat_result.created_visualizations else [] + if not objects: + return "no visualization was created" + want_metrics = {_strip_type(m) for m in expected.metrics} + want_attributes = {_strip_type(a) for a in expected.attributes} + seen: list[dict[str, list[str]]] = [] + for viz in objects: + metrics = {_strip_type(u) for u in get_metric_uri_set(viz)} + attributes = {_strip_type(u) for u in get_dimension_uri_set(viz)} + if want_metrics <= metrics and want_attributes <= attributes: + return None + seen.append({"metrics": sorted(metrics), "attributes": sorted(attributes)}) + return f"created {seen}" + + +def _set_check( + record: TurnRecord, run: AgentSwitcherRunResult, name: str, ok: bool, expected: Any, actual: Any +) -> None: + """Record one per-turn check on the turn and, as `turn_`, on the run.""" + record.checks[name] = ok + full = f"turn{record.turn}_{name}" + if ok: + run.checks[full] = True + else: + run.fail(FailedCheck(full, expected=expected, actual=actual)) + + +def _turn_code_checks( + record: TurnRecord, run: AgentSwitcherRunResult, turn: AgentSwitcherTurn, result: ChatResult +) -> None: + """The turn's checks read from its chat result.""" + checks = turn.checks + if checks.alert_metric is not None: + ids = _proposal_metric_ids(result) + ok = _strip_type(checks.alert_metric) in {_strip_type(i) for i in ids} + _set_check(record, run, "alert_metric", ok, checks.alert_metric, ids) + if checks.visualization is not None: + problem = _visualization_problem(checks.visualization, result) + _set_check(record, run, "visualization", problem is None, checks.visualization.model_dump(), problem) + + +def _skills_checks( + record: TurnRecord, run: AgentSwitcherRunResult, turn: AgentSwitcherTurn, unreadable: str | None +) -> None: + """`skills_used` / `skills_not_used` against `record.skills`; both fail when the items were unreadable.""" + checks = turn.checks + if checks.skills_used: + ok = unreadable is None and set(checks.skills_used) <= set(record.skills) + _set_check(record, run, "skills_used", ok, checks.skills_used, unreadable or record.skills) + if checks.skills_not_used: + used = sorted(set(checks.skills_not_used) & set(record.skills)) + ok = unreadable is None and not used + _set_check(record, run, "skills_not_used", ok, f"none of {checks.skills_not_used}", unreadable or used) + + +def _skills_in_use(item: dict[str, Any]) -> list[str]: + """`skillsInUse[].name` of a toolCall item; empty for any other item.""" + content = item.get("content") + if item.get("role") != "assistant" or not isinstance(content, dict) or content.get("type") != "toolCall": + return [] + refs = item.get("skillsInUse") + names = (ref.get("name") for ref in refs if isinstance(ref, dict)) if isinstance(refs, list) else () + return [n for n in names if isinstance(n, str)] + + +def _switch_error(turn: int, agent_id: str, exc: httpx.HTTPError) -> str: + head = f"switchAgent to {agent_id!r} before turn {turn} failed" + if not isinstance(exc, httpx.HTTPStatusError): + return f"{head}: {type(exc).__name__}: {exc}" + status = exc.response.status_code + if status == 404: + return f"{head} with 404: the agent-switching feature flag may be off, or the agent is unknown" + if status == 403: + return f"{head} with 403: no access to the agent" + return f"{head} with {status}: {exc.response.text[:200]}" + + +def _check_items( + run: AgentSwitcherRunResult, + spec: AgentSwitcherInput, + items: list[dict[str, Any]], + expected_markers: list[tuple[int, str]], +) -> None: + """Skills checks, `agent_stamped` and `switch_marker` from the stored items. + + - a user item opens a turn; the turn runs up to the next user item; + - a switch marker (`trigger` `agent_switched`) counts by the user items before it, so the + marker for the switch before turn N comes after N-1 user items; + - fills each turn's answering agents and used skills. + """ + user_items = 0 + answering: dict[int, list[str]] = {} + skills: dict[int, list[str]] = {} + markers: list[tuple[int, str]] = [] + for item in items: + role = item.get("role") + if role == "user": + user_items += 1 + elif role == "assistant": + answering.setdefault(user_items, []).append(str(item.get("agentId"))) + turn_skills = skills.setdefault(user_items, []) + turn_skills.extend(n for n in _skills_in_use(item) if n not in turn_skills) + elif role == "system" and item.get("trigger") == "agent_switched": + markers.append((user_items, str(item.get("agentId") or item.get("newAgentId")))) + + for record, turn in zip(run.turns, spec.turns, strict=False): + record.skills = skills.get(record.turn, []) + _skills_checks(record, run, turn, unreadable=None) + + stamping: list[str] = [] + if user_items != len(run.turns): + stamping.append(f"{user_items} user item(s) for {len(run.turns)} turn(s)") + if answering.get(0): + stamping.append(f"assistant item(s) before the first user item: {answering[0]}") + for record in run.turns: + agents = answering.get(record.turn, []) + record.answering_agents = list(dict.fromkeys(agents)) + if not agents: + stamping.append(f"turn {record.turn}: no assistant item") + elif record.answering_agents != [record.agent_id]: + stamping.append(f"turn {record.turn}: {record.answering_agents}, active {record.agent_id!r}") + if stamping: + expected = "every assistant item stamped with the active agent" + run.fail(FailedCheck("agent_stamped", expected=expected, actual=stamping)) + else: + run.checks["agent_stamped"] = True + + if markers == expected_markers: + run.checks["switch_marker"] = True + else: + run.fail( + FailedCheck( + "switch_marker", + expected=[{"after_user_items": n, "agent_id": a} for n, a in expected_markers], + actual=[{"after_user_items": n, "agent_id": a} for n, a in markers], + ) + ) + + +def _transcript(turns: list[TurnRecord], current: TurnRecord) -> str: + """The conversation before `current`, then its message, for the judge's INPUT.""" + lines = ["CONVERSATION BEFORE:"] if turns else ["CONVERSATION BEFORE: (none)"] + for record in turns: + agent = ", ".join(record.answering_agents) or record.agent_id + answer = " ".join(record.answer.split()) + if len(answer) > _TRANSCRIPT_ANSWER_CHARS: + answer = answer[: _TRANSCRIPT_ANSWER_CHARS - 1] + "…" + lines.append(f"USER: {record.message}") + lines.append(f"ASSISTANT (agent {agent}): {answer}") + current_agent = ", ".join(current.answering_agents) or current.agent_id + lines.append(f"LATEST MESSAGE: {current.message}") + lines.append(f"ANSWERED BY: agent {current_agent}") + return "\n".join(lines) + + +def _judge_turns(run: AgentSwitcherRunResult, spec: AgentSwitcherInput, judges: _Judges) -> None: + """Grade every turn's criteria; an ungraded criterion leaves the run's verdict open.""" + ungraded: list[str] = [] + for index, (record, turn) in enumerate(zip(run.turns, spec.turns, strict=True)): + if not turn.checks.judged: + continue + must_include: list[Criterion] = [(f"include_{i}", c) for i, c in enumerate(turn.checks.must_include)] + must_not_include: list[Criterion] = [(f"exclude_{i}", c) for i, c in enumerate(turn.checks.must_not_include)] + detail: dict[str, Any] = {} + grade_rubric( + judges.positive, + judges.violation, + question=_transcript(run.turns[:index], record), + actual=record.answer, + must_include=must_include, + must_not_include=must_not_include, + rubric=[], + detail=detail, + ) + for key, criterion in [*must_include, *must_not_include]: + reason = str(detail.get(f"{key}_reason", "")) + record.reasons[key] = reason + if key not in detail: + ungraded.append(f"turn {record.turn} {key}: {reason}") + continue + ok = detail[key] is True + record.checks[key] = ok + name = f"turn{record.turn}_{key}" + if ok: + run.checks[name] = True + else: + run.fail(FailedCheck(name, part="judge", expected=criterion, actual=reason)) + if ungraded and all(run.checks.values()): + run.judge_error = " | ".join(ungraded) + + +def _run_once( + client: ChatClient, spec: AgentSwitcherInput, judges: _Judges | None, conversation_id: str +) -> AgentSwitcherRunResult: + started = time.monotonic() + run = AgentSwitcherRunResult(conversation_id=conversation_id) + active = spec.agent_id + # (user items before the marker, new agent) for every switch that changes the agent. + expected_markers: list[tuple[int, str]] = [] + agent_s = 0.0 + offsets = (0.0, 0, 0) + + for number, turn in enumerate(spec.turns, start=1): + if turn.switch_to is not None: + try: + conversation = client.switch_agent(conversation_id, turn.switch_to) + except httpx.HTTPError as exc: + run.error = _switch_error(number, turn.switch_to, exc) + run.exit_reason = LoopExit.CHAT_ERROR + run.fail(FailedCheck(f"turn{number}_switched", expected=turn.switch_to, actual=run.error)) + break + bound = conversation.get("agentId") if isinstance(conversation, dict) else None + if bound != turn.switch_to: + run.error = f"switchAgent before turn {number} answered agentId {bound!r}, asked for {turn.switch_to!r}" + run.exit_reason = LoopExit.CHAT_ERROR + run.fail(FailedCheck(f"turn{number}_switched", expected=turn.switch_to, actual=bound)) + break + if turn.switch_to != active: + expected_markers.append((number - 1, turn.switch_to)) + active = turn.switch_to + + record = TurnRecord(turn=number, message=turn.message, agent_id=active, switch_to=turn.switch_to) + sent = time.monotonic() + try: + result = client.send_message(conversation_id, turn.message) + except ChatError as exc: + agent_s += time.monotonic() - sent + partial = exc.partial_result + if partial is not None: + run.reasoning_steps.extend(partial.reasoning_steps or []) + run.response_id = partial.response_id or run.response_id + run.error = f"turn {number}: {type(exc).__name__}: {exc}" + run.exit_reason = chat_failure_exit(exc) + run.fail(FailedCheck(f"turn{number}_answered", actual=run.error)) + run.turns.append(record) + break + agent_s += time.monotonic() - sent + record.answer = render_turn_for_simulated_user(result) + run.reasoning_steps.extend(result.reasoning_steps or []) + run.response_id = result.response_id or run.response_id + offsets = shift_and_index_events( + result, turn_offset=offsets[0], tool_index_offset=offsets[1], reasoning_index_offset=offsets[2] + ) + run.tool_call_events.extend(result.tool_call_events or []) + run.reasoning_step_events.extend(result.reasoning_step_events or []) + run.turns.append(record) + _turn_code_checks(record, run, turn, result) + + judge_s = 0.0 + if run.error is None: + try: + items = client.list_items(conversation_id) + except (httpx.HTTPError, ValueError) as exc: + run.error = f"reading the conversation items failed: {type(exc).__name__}: {exc}" + run.exit_reason = LoopExit.CHAT_ERROR + run.fail(FailedCheck("items_read", actual=run.error)) + for record, turn in zip(run.turns, spec.turns, strict=False): + _skills_checks(record, run, turn, unreadable="items unreadable") + else: + _check_items(run, spec, items, expected_markers) + if judges is not None: + judged = time.monotonic() + _judge_turns(run, spec, judges) + judge_s = time.monotonic() - judged + + run.timings = PhaseTimings(agent_s=agent_s, judge_s=judge_s) + run.run_latency_s = time.monotonic() - started + return run + + +def run_agentic_agent_switcher( + host: str, + token: str, + workspace_id: str, + spec: AgentSwitcherInput, + k: int = _DEFAULT_K, + reasoning_effort: ReasoningEffort | None = None, + user_context: dict | None = None, +) -> AgenticAgentSwitcherSummary: + """Replay the item K times, each run on a conversation of its own, and return a summary. + + Raises: + ValueError: ``k`` is below 1. + """ + if k < 1: + raise ValueError(f"k must be at least 1, got {k}") + # Built before any request: a missing judge key fails the item, not its runs. + judges = ( + _Judges(LLMJudge(_POSITIVE_STEPS), LLMJudge(_VIOLATION_STEPS)) + if any(t.checks.judged for t in spec.turns) + else None + ) + client = ChatClient( + host=host, + token=token, + workspace_id=workspace_id, + reasoning_effort=reasoning_effort, + agent_id=spec.agent_id, + user_context=user_context, + ) + run_results: list[AgentSwitcherRunResult] = [] + try: + for _ in range(k): + conversation_id = client.create_conversation() + try: + run_results.append(_run_once(client, spec, judges, conversation_id)) + finally: + client.delete_conversation(conversation_id) + finally: + client.close() + + # As general_question: pass@K ignores ungraded runs, pass^K is not met by one. + scored = [r for r in run_results if r.judge_error is None] + pass_at_k = any(r.passed for r in scored) + pass_power_k = len(scored) == len(run_results) and all(r.passed for r in scored) + best = max(scored or run_results, key=lambda r: (r.passed, sum(r.checks.values()))) + return AgenticAgentSwitcherSummary(run_results, pass_at_k, pass_power_k, best) + + +def _kind_fields(run: AgentSwitcherRunResult) -> dict[str, Any]: + return { + **run.checks, + "turns": [t.as_record() for t in run.turns], + **({"error": run.error} if run.error is not None else {}), + } + + +def _run_detail(run: AgentSwitcherRunResult) -> dict[str, Any]: + return { + **_kind_fields(run), + "latency_breakdown": build_latency_breakdown(run.tool_call_events, run.reasoning_step_events), + } + + +def _failed_checks(run: AgentSwitcherRunResult) -> list[dict]: + return failed_check_records(run.failed) + + +def _attach_diagnostics( + error: AgenticAssertionError | JudgeResponseError, + best: AgentSwitcherRunResult, + detail: dict, + failed_runs: list[dict], + runs_passed: int, + runs_effective: int, + run_latencies_s: list[float], +) -> None: + error.reasoning_steps = best.reasoning_steps + error.conversation_id = best.conversation_id + error.response_id = best.response_id + error.detail = detail + error.runs_passed = runs_passed + error.runs_effective = runs_effective + error.failed_runs = failed_runs + error.run_latencies_s = run_latencies_s + error.best_run_latency_s = best.run_latency_s + + +def _dataset_item(dataset_item: DatasetItem | dict, dataset_name: str) -> tuple[DatasetItem, AgentSwitcherInput]: + if isinstance(dataset_item, DatasetItem): + item = dataset_item + else: + item = DatasetItem.model_validate( + {"id": "", "test_kind": _TEST_KIND, "expected_output": {}, **dataset_item, "dataset_name": dataset_name} + ) + if item.agent_switcher_input is None: + raise ValueError(f"{_TEST_KIND} item '{item.id}' is missing required 'agent_switcher_input'.") + return item, item.agent_switcher_input + + +def evaluate_agentic_agent_switcher( + host: str, + token: str, + workspace_id: str, + dataset_item: DatasetItem | dict, + k: int = _DEFAULT_K, + langfuse: object | None = None, + dataset_item_id: str | None = None, + dataset_name: str = "agent_switcher", + run_timestamp: str | None = None, + model_version_override: str | None = None, + run_metadata_extra: dict | None = None, + reasoning_effort: ReasoningEffort | None = None, + submit_trace_link: SubmitTraceLink = run_trace_link_inline, + gate: EvalGate = DEFAULT_GATE, +) -> AgenticEvalOutcome: + """Run the agent-switcher evaluation K times, log to Langfuse, and raise on a failed gate. + + Args: + host: GoodData host URL. + token: API token. + workspace_id: Workspace the conversations run in. + dataset_item: The item as stored, carrying `agent_switcher_input` and optionally `user_context`. + k: Runs per item; each creates its own conversation. + langfuse: Langfuse client; built from the LANGFUSE_* environment when None. + dataset_item_id: Langfuse item id; the item's own `id` when None. + dataset_name: Langfuse dataset name; prefixes the experiment run name. + run_timestamp: Shared run timestamp for the run name. + model_version_override: Model-version tag for the run. + run_metadata_extra: Extra experiment-run metadata. + reasoning_effort: Sent on every message. + submit_trace_link: How the Langfuse block is run; inline by default. + gate: `any` (pass@K) or `power` (pass^K). + + Raises: + AgentSwitcherAssertionError: the gate did not pass. + JudgeResponseError: the judge returned no readable verdict for any run. + ValueError: the item has no valid `agent_switcher_input`, raised before any request. + """ + item, spec = _dataset_item(dataset_item, dataset_name) + item_id = item.id if dataset_item_id is None else dataset_item_id + identity = RunIdentity( + host, + token, + workspace_id, + dataset_name, + run_timestamp, + model_version_override, + run_metadata_extra, + reasoning_effort, + ) + langfuse, window_start, scope = open_item_trace(langfuse, identity, item_id, suffix_runs=k > 1) + with item_scope(scope): + summary = run_agentic_agent_switcher( + host=host, + token=token, + workspace_id=workspace_id, + spec=spec, + k=k, + reasoning_effort=reasoning_effort, + user_context=item.user_context, + ) + + run_labels = label_runs(summary.run_results, passed=lambda r: r.passed) + + if langfuse is not None and item_id: + window_end = utc_now() + + def _write_scores(ctx: RunTraceContext) -> None: + stamp_gate_metadata(ctx.run_metadata, k=len(summary.run_results), gate=gate) + for run_idx, run in enumerate(summary.run_results): + if run.judge_error is not None: + mark_unscored_run(ctx, run_idx, run.conversation_id, run_labels.error_label(run_idx)) + continue + pt = ctx.trace(run.conversation_id) + turn_checks = [ok for name, ok in run.checks.items() if name.startswith("turn")] + with ctx.observe( + pt, run_idx, conversation_id=run.conversation_id, output={"agent_switcher_pass": run.passed} + ) as tid: + ctx.score(tid, name="agent_switcher_pass", value=float(run.passed), data_type="BOOLEAN") + for name in ("agent_stamped", "switch_marker"): + if name in run.checks: + ctx.score(tid, name=name, value=float(run.checks[name]), data_type="BOOLEAN") + if turn_checks: + ctx.score(tid, name="turn_checks_pass", value=float(all(turn_checks)), data_type="BOOLEAN") + log_gate_scores(ctx, tid, gate=gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k) + ctx.quality( + tid, + strict_checks=run.checks, + latency_sec=pt.latency if pt else None, + cost_usd=pt.total_cost if pt else None, + ) + + submit_trace_scoring( + submit_trace_link, + identity, + langfuse=langfuse, + dataset_item_id=item_id, + conversation_ids=conversations_to_link( + [r.conversation_id for r in summary.scored_run_results], + [r.conversation_id for r in summary.run_results if r.judge_error is not None], + ), + window_start=window_start, + window_end=window_end, + suffix_runs=k > 1, + write_scores=_write_scores, + scope=scope, + item_input=item.question, + ) + + item_timings = sum_timings([r.timings for r in summary.run_results]) + unscored = summary.judge_errors + runs_passed = sum(1 for r in summary.scored_run_results if r.passed) + runs_effective = len(summary.run_results) + run_latencies_s = [r.run_latency_s for r in summary.run_results] + best = summary.best + detail = agentic_detail( + best.tool_call_events, + best.reasoning_step_events, + **_kind_fields(best), + **({"unscored_runs": len(unscored), "judge_errors": unscored} if unscored else {}), + ) + failed_runs = build_failed_runs( + summary.run_results, labels=run_labels, detail=_run_detail, failed_checks=_failed_checks + ) + + if not summary.scored_run_results: + error = JudgeResponseError( + f"judge returned no readable verdict for any of the {len(summary.run_results)} run(s): " + + " | ".join(unscored) + ) + _attach_diagnostics(error, best, detail, failed_runs, runs_passed, runs_effective, run_latencies_s) + error.timings = item_timings + attach_labels(error, run_labels, gate) + raise error + + if not gate_passed(gate, pass_at_k=summary.pass_at_k, pass_power_k=summary.pass_power_k): + head = failure_message_head(gate, run_labels, runs_passed, failed_runs, len(unscored)) + raise_agentic_failure( + AgentSwitcherAssertionError, + f"Agent switcher assertion failed. {head}" + (f" Best run error: {best.error}" if best.error else ""), + harness_error=run_labels.harness_error_record, + shadow=run_labels.shadow(gate), + reasoning_steps=best.reasoning_steps, + conversation_id=best.conversation_id, + response_id=best.response_id, + detail=detail, + runs_passed=runs_passed, + runs_effective=runs_effective, + failed_runs=failed_runs, + run_latencies_s=run_latencies_s, + best_run_latency_s=best.run_latency_s, + timings=item_timings, + ) + return agentic_success( + shadow=run_labels.shadow(gate), + reasoning_steps=best.reasoning_steps, + conversation_id=best.conversation_id, + response_id=best.response_id, + detail=detail, + runs_passed=runs_passed, + runs_effective=runs_effective, + failed_runs=failed_runs, + run_latencies_s=run_latencies_s, + best_run_latency_s=best.run_latency_s, + timings=item_timings, + ) diff --git a/packages/gooddata-eval/src/gooddata_eval/core/chat/sse_client.py b/packages/gooddata-eval/src/gooddata_eval/core/chat/sse_client.py index 78a7388aa..87f5e72c2 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/chat/sse_client.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/chat/sse_client.py @@ -554,6 +554,38 @@ def _do() -> str: self._conversation_started = time.monotonic() return conversation_id + def switch_agent(self, conversation_id: str, agent_id: str) -> dict[str, Any]: + """Bind the conversation to another agent and return the conversation. + + gen-ai answers a switch to the agent already bound as a no-op, so a retry is safe. + A 404 means the agent-switching flag is off or the agent is unknown; 403, no access. + """ + + def _do() -> dict[str, Any]: + resp = self._client.post( + f"{self._base}/{conversation_id}/switchAgent", + headers={**self._auth, "Content-Type": "application/json"}, + json={"agentId": agent_id}, + ) + resp.raise_for_status() + return resp.json() + + return retry_transient(_do, is_retryable=_is_retryable_exc) + + def list_items(self, conversation_id: str) -> list[dict[str, Any]]: + """Every stored item of the conversation, in order, as gen-ai returns them (camelCase).""" + + def _do() -> list[dict[str, Any]]: + resp = self._client.get(f"{self._base}/{conversation_id}/items", headers=self._auth) + resp.raise_for_status() + body = resp.json() + items = body.get("items") if isinstance(body, dict) else None + if not isinstance(items, list): + raise ValueError(f"GoodData /items response missing 'items' list: {body}") + return items + + return retry_transient(_do, is_retryable=_is_retryable_exc) + def delete_conversation(self, conversation_id: str) -> None: try: self._client.delete(f"{self._base}/{conversation_id}", headers=self._auth) diff --git a/packages/gooddata-eval/src/gooddata_eval/core/dataset/langfuse_source.py b/packages/gooddata-eval/src/gooddata_eval/core/dataset/langfuse_source.py index 878bac8e7..f4ac0dd90 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/dataset/langfuse_source.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/dataset/langfuse_source.py @@ -18,7 +18,7 @@ from pydantic import ValidationError from gooddata_eval.core.langfuse._env import make_http_client -from gooddata_eval.core.models import CatalogInput, DatasetItem, SummaryInput, TakeawayInput +from gooddata_eval.core.models import AgentSwitcherInput, CatalogInput, DatasetItem, SummaryInput, TakeawayInput _PAGE_SIZE = 100 @@ -102,6 +102,28 @@ def _takeaway_input_from_raw(raw: dict, expected_output: Any) -> TakeawayInput | return None +def _agent_switcher_input_from_raw(raw: dict, expected_output: Any) -> AgentSwitcherInput | None: + """Locate an agentic_agent_switcher item's `agent_switcher_input`, the same way as `takeaway_input`. + + Raises: + ValueError: The value found is not a valid `agent_switcher_input` object. + """ + for source in (raw.get("input"), raw.get("metadata"), expected_output): + if not isinstance(source, dict) or source.get("agent_switcher_input") is None: + continue + found = source["agent_switcher_input"] + if not isinstance(found, dict): + raise ValueError( + f"Langfuse item {raw.get('id')!r}: agent_switcher_input must be a JSON object, " + f"got {type(found).__name__}" + ) + try: + return AgentSwitcherInput.model_validate(found) + except ValidationError as e: + raise ValueError(f"Langfuse item {raw.get('id')!r}: invalid agent_switcher_input: {e}") from e + return None + + def _user_context_from_raw(raw: dict) -> dict[str, Any] | None: """Locate an item's `user_context` -- relayed verbatim as the chat request's `userContext`. @@ -160,13 +182,17 @@ def _item_from_raw(raw: dict, *, dataset_name: str, test_kind: str) -> DatasetIt expected_output = raw.get("expectedOutput") or raw.get("expected_output") takeaway_input = _takeaway_input_from_raw(raw, expected_output) catalog_input = _catalog_input_from_raw(raw, expected_output) - # An item carrying `takeaway_input` or `catalog_input` can only be of that kind, so it stands - # in for the CLI default; an explicit test_kind or a structural match still wins. + agent_switcher_input = _agent_switcher_input_from_raw(raw, expected_output) + # An item carrying `takeaway_input`, `catalog_input` or `agent_switcher_input` can only be of + # that kind, so it stands in for the CLI default; an explicit test_kind or a structural match + # still wins. default_kind = test_kind if catalog_input is not None: default_kind = "catalog_metadata" if takeaway_input is not None: default_kind = "document_takeaway" + if agent_switcher_input is not None: + default_kind = "agentic_agent_switcher" resolved_kind = _infer_test_kind(expected_output, default_kind, raw.get("metadata")) return DatasetItem( id=str(raw["id"]), @@ -177,6 +203,7 @@ def _item_from_raw(raw: dict, *, dataset_name: str, test_kind: str) -> DatasetIt summary_input=_summary_input_from_raw(raw, expected_output), takeaway_input=takeaway_input, catalog_input=catalog_input, + agent_switcher_input=agent_switcher_input, user_context=_user_context_from_raw(raw), ) diff --git a/packages/gooddata-eval/src/gooddata_eval/core/interaction.py b/packages/gooddata-eval/src/gooddata_eval/core/interaction.py index 645625697..b95727955 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/interaction.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/interaction.py @@ -46,6 +46,7 @@ "agentic_anomaly_detection", "agentic_what_if", "agentic_forecasting", + "agentic_agent_switcher", } ) diff --git a/packages/gooddata-eval/src/gooddata_eval/core/models.py b/packages/gooddata-eval/src/gooddata_eval/core/models.py index eceda9f54..c1d920215 100644 --- a/packages/gooddata-eval/src/gooddata_eval/core/models.py +++ b/packages/gooddata-eval/src/gooddata_eval/core/models.py @@ -579,6 +579,55 @@ class CatalogInput(BaseModel): output: Literal["title", "description"] +class AgentSwitcherVisualization(BaseModel): + """Fields a turn's created visualization must use; a subset match, ids with or without type prefix.""" + + model_config = ConfigDict(extra="forbid") + + metrics: list[str] = Field(default_factory=list) + attributes: list[str] = Field(default_factory=list) + + +class AgentSwitcherChecks(BaseModel): + """What one turn of an `agentic_agent_switcher` item must show. Every key is optional.""" + + model_config = ConfigDict(extra="forbid") + + must_include: list[str] = Field(default_factory=list) + must_not_include: list[str] = Field(default_factory=list) + skills_used: list[str] = Field(default_factory=list) + skills_not_used: list[str] = Field(default_factory=list) + alert_metric: str | None = Field(default=None, min_length=1) + visualization: AgentSwitcherVisualization | None = None + + @property + def judged(self) -> bool: + return bool(self.must_include or self.must_not_include) + + +class AgentSwitcherTurn(BaseModel): + """One user message; `switch_to` switches the agent before it is sent.""" + + model_config = ConfigDict(extra="forbid") + + message: str = Field(min_length=1) + switch_to: str | None = Field(default=None, min_length=1) + checks: AgentSwitcherChecks = Field(default_factory=AgentSwitcherChecks) + + +class AgentSwitcherInput(BaseModel): + """Structured input for the `agentic_agent_switcher` test kind. + + `agent_id` is the agent the conversation is created with. Unknown keys are rejected at + every level so a typo in a dataset item fails loudly. + """ + + model_config = ConfigDict(extra="forbid") + + agent_id: str = Field(min_length=1) + turns: list[AgentSwitcherTurn] = Field(min_length=1) + + class DatasetItem(BaseModel): """Common dataset envelope. `expected_output` stays raw; each evaluator parses its own shape.""" @@ -595,6 +644,8 @@ class DatasetItem(BaseModel): takeaway_input: TakeawayInput | None = None # Only used by the `catalog_metadata` test kind; ignored by all others. catalog_input: CatalogInput | None = None + # Only used by the `agentic_agent_switcher` test kind; ignored by all others. + agent_switcher_input: AgentSwitcherInput | None = None # Relayed verbatim as the chat request's `userContext`. Deliberately opaque: gen-ai owns # that schema (a discriminated union of view/widget descriptors), so re-modelling it here # would only create a second copy to keep in sync. diff --git a/packages/gooddata-eval/tests/test_agentic_agent_switcher.py b/packages/gooddata-eval/tests/test_agentic_agent_switcher.py new file mode 100644 index 000000000..a3b2b2a62 --- /dev/null +++ b/packages/gooddata-eval/tests/test_agentic_agent_switcher.py @@ -0,0 +1,526 @@ +# (C) 2026 GoodData Corporation. All rights reserved. +# SPDX-License-Identifier: LicenseRef-GoodData-Enterprise +import contextlib +import json +from typing import Any +from unittest.mock import MagicMock, call, patch + +import httpx +import pytest +from gooddata_eval.cli.agentic_runner import AGENTIC_TEST_KINDS, _dispatch_agentic +from gooddata_eval.core.agentic.agent_switcher import ( + AgentSwitcherAssertionError, + evaluate_agentic_agent_switcher, + run_agentic_agent_switcher, +) +from gooddata_eval.core.chat.sse_client import ChatError +from gooddata_eval.core.dataset.langfuse_source import _item_from_raw +from gooddata_eval.core.interaction import MULTI_TURN_KINDS +from gooddata_eval.core.models import AgentSwitcherInput, ChatResult, DatasetItem +from pydantic import ValidationError + +_MODULE = "gooddata_eval.core.agentic.agent_switcher" + +_TWO_TURNS: dict[str, Any] = { + "agent_id": "planner", + "turns": [ + {"message": "Plan: watch revenue by region weekly."}, + {"message": "Now set it up.", "switch_to": "builder"}, + ], +} + + +def _item(spec: dict[str, Any] | None = None, **extra: Any) -> dict[str, Any]: + return { + "id": "switch-1", + "question": "Plan with one agent, build with another", + "agent_switcher_input": spec or _TWO_TURNS, + **extra, + } + + +def _chat( + text: str = "ok", + *, + skills: list[str] | None = None, + proposals: list[dict] | None = None, + viz: dict | None = None, +) -> ChatResult: + events: list[dict[str, Any]] = ( + [{"functionName": "set_skills", "functionArguments": json.dumps({"skill_names": skills})}] + if skills is not None + else [] + ) + return ChatResult.model_validate( + { + "textResponse": text, + "toolCallEvents": events, + "alertProposals": proposals or [], + "createdVisualizations": {"objects": [viz]} if viz else None, + } + ) + + +def _tool_call(agent: str, name: str, skills: list[str]) -> dict[str, Any]: + refs = [{"name": s, "title": s.title()} for s in skills] + return {"role": "assistant", "agentId": agent, "content": {"type": "toolCall", "name": name}, "skillsInUse": refs} + + +def _items(*turns: tuple[Any, ...]) -> list[dict[str, Any]]: + """Stored items for turns given as (marker agent or None, answering agent[, skills whose tools ran]).""" + items: list[dict[str, Any]] = [] + for marker, agent, *rest in turns: + if marker is not None: + items.append({"role": "system", "agentId": marker, "newAgentId": marker, "trigger": "agent_switched"}) + items.append({"role": "user", "agentId": agent, "skillsInUse": None}) + items.append(_tool_call(agent, "set_skills", [])) + for skill in rest[0] if rest else []: + items.append(_tool_call(agent, f"{skill}_tool", [skill])) + items.append({"role": "tool", "agentId": agent, "content": {"type": "toolResult"}}) + items.append({"role": "assistant", "agentId": agent, "content": {"type": "text"}, "skillsInUse": None}) + return items + + +def _client(answers: list[ChatResult], items: list[dict[str, Any]], *, conversations: int = 1) -> MagicMock: + client = MagicMock() + client.create_conversation.side_effect = [f"conv-{i}" for i in range(1, conversations + 1)] + client.send_message.side_effect = answers + client.switch_agent.side_effect = lambda _conv, agent: {"conversationId": _conv, "agentId": agent} + client.list_items.return_value = items + return client + + +def _judge(verdict: bool = True) -> MagicMock: + judge = MagicMock() + judge.model = "gpt-4o" + judge.score.return_value = (verdict, "because") + return judge + + +@contextlib.contextmanager +def _patched(client: MagicMock, positive: MagicMock | None = None): + """The positive judge passes unless given; the violation judge finds nothing.""" + judges = [positive or _judge(), _judge(verdict=False)] + with ( + patch(f"{_MODULE}.ChatClient", return_value=client), + patch(f"{_MODULE}.LLMJudge", side_effect=judges) as mock_judge, + ): + yield mock_judge + + +def _evaluate(client: MagicMock, item: dict[str, Any] | None = None, positive: MagicMock | None = None, **kwargs: Any): + with _patched(client, positive): + return evaluate_agentic_agent_switcher("https://h", "tok", "ws1", item or _item(), **kwargs) + + +def _failed_check_names(exc: AgentSwitcherAssertionError) -> list[str]: + return [c["name"] for run in exc.failed_runs for c in run["failed_checks"]] + + +# --------------------------------------------------------------------------- # +# item validation +# --------------------------------------------------------------------------- # +@pytest.mark.parametrize( + "spec", + [ + {"agent_id": "a", "turns": []}, + {"agent_id": "", "turns": [{"message": "hi"}]}, + {"turns": [{"message": "hi"}]}, + {"agent_id": "a", "turns": [{"message": "hi", "switch_too": "b"}]}, + {"agent_id": "a", "turns": [{"message": "hi", "checks": {"must_includes": ["x"]}}]}, + {"agent_id": "a", "turns": [{"message": "hi", "checks": {"visualization": {"metric": ["x"]}}}]}, + {"agent_id": "a", "turns": [{"message": "hi", "switch_to": ""}]}, + {"agent_id": "a", "extra": 1, "turns": [{"message": "hi"}]}, + ], +) +def test_an_invalid_input_is_rejected_when_the_item_loads(spec): + with pytest.raises(ValidationError): + DatasetItem.model_validate({"id": "i", "dataset_name": "d", "test_kind": "t", "question": "q", **_item(spec)}) + + +def test_an_item_without_input_fails_before_any_request(): + client = _client([], []) + with pytest.raises(ValueError, match="agent_switcher_input"), _patched(client): + evaluate_agentic_agent_switcher("https://h", "tok", "ws1", {"id": "i", "question": "q"}) + client.create_conversation.assert_not_called() + + +def test_the_langfuse_loader_reads_the_input_and_infers_the_kind(): + raw = {"id": "lf-1", "input": {"question": "q"}, "metadata": {"agent_switcher_input": _TWO_TURNS}} + item = _item_from_raw(raw, dataset_name="ds", test_kind="visualization") + assert item.test_kind == "agentic_agent_switcher" + assert item.agent_switcher_input == AgentSwitcherInput.model_validate(_TWO_TURNS) + + +def test_the_kind_is_registered_and_multi_turn(): + assert "agentic_agent_switcher" in AGENTIC_TEST_KINDS + assert "agentic_agent_switcher" in MULTI_TURN_KINDS + + +def test_dispatch_hands_the_whole_item_over(): + item = DatasetItem.model_validate( + {"dataset_name": "d", "test_kind": "agentic_agent_switcher", "expected_output": {}, **_item()} + ) + with patch("gooddata_eval.cli.agentic_runner.evaluate_agentic_agent_switcher") as mock_eval: + _dispatch_agentic(item, "https://h", "tok", "ws1", 3, None, "ts", None, gate="power") + kwargs = mock_eval.call_args.kwargs + assert kwargs["dataset_item"] is item + assert (kwargs["k"], kwargs["gate"], kwargs["dataset_item_id"]) == (3, "power", "switch-1") + + +# --------------------------------------------------------------------------- # +# switching and the item checks +# --------------------------------------------------------------------------- # +def test_switches_before_the_turn_that_asks_for_it(): + client = _client([_chat(), _chat()], _items((None, "planner"), ("builder", "builder"))) + outcome = _evaluate(client) + + calls = [c for c in client.mock_calls if c[0] in ("create_conversation", "send_message", "switch_agent")] + assert calls == [ + call.create_conversation(), + call.send_message("conv-1", "Plan: watch revenue by region weekly."), + call.switch_agent("conv-1", "builder"), + call.send_message("conv-1", "Now set it up."), + ] + client.list_items.assert_called_once_with("conv-1") + client.delete_conversation.assert_called_once_with("conv-1") + assert outcome.runs_passed == 1 + assert outcome.detail["agent_stamped"] is True + assert outcome.detail["switch_marker"] is True + assert [t["answering_agents"] for t in outcome.detail["turns"]] == [["planner"], ["builder"]] + + +def test_the_client_starts_on_the_item_agent_with_its_context(): + client = _client([_chat(), _chat()], _items((None, "planner"), ("builder", "builder"))) + with patch(f"{_MODULE}.ChatClient", return_value=client) as mock_client: + evaluate_agentic_agent_switcher("https://h", "tok", "ws1", _item(user_context={"view": {}})) + assert mock_client.call_args.kwargs["agent_id"] == "planner" + assert mock_client.call_args.kwargs["user_context"] == {"view": {}} + + +def test_an_answer_stamped_with_the_previous_agent_fails_agent_stamped(): + client = _client([_chat(), _chat()], _items((None, "planner"), ("builder", "planner"))) + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client) + assert _failed_check_names(exc.value) == ["agent_stamped"] + assert "turn 2" in str(exc.value) + + +def test_a_missing_marker_fails_switch_marker(): + client = _client([_chat(), _chat()], _items((None, "planner"), (None, "builder"))) + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client) + assert _failed_check_names(exc.value) == ["switch_marker"] + + +def test_a_marker_carrying_the_old_agent_fails_switch_marker(): + client = _client([_chat(), _chat()], _items((None, "planner"), ("planner", "builder"))) + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client) + assert _failed_check_names(exc.value) == ["switch_marker"] + + +def test_a_system_item_without_the_switch_trigger_is_not_a_marker(): + items = _items((None, "planner"), ("builder", "builder")) + items.insert(1, {"role": "system", "agentId": "planner"}) + client = _client([_chat(), _chat()], items) + assert _evaluate(client).runs_passed == 1 + + +def test_a_drift_marker_is_not_counted_as_a_switch(): + items = _items((None, "planner"), ("builder", "builder")) + items.insert(0, {"role": "system", "agentId": "planner", "trigger": "agent_updated"}) + client = _client([_chat(), _chat()], items) + assert _evaluate(client).runs_passed == 1 + + +def test_a_switch_to_the_bound_agent_expects_no_marker(): + spec = {"agent_id": "planner", "turns": [{"message": "a"}, {"message": "b", "switch_to": "planner"}]} + client = _client([_chat(), _chat()], _items((None, "planner"), (None, "planner"))) + assert _evaluate(client, _item(spec)).runs_passed == 1 + + +def test_a_404_on_switch_fails_the_run_and_names_the_flag(): + client = _client([_chat()], []) + request = httpx.Request("POST", "https://h/switchAgent") + client.switch_agent.side_effect = httpx.HTTPStatusError( + "404", request=request, response=httpx.Response(404, request=request) + ) + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client) + assert "flag may be off" in str(exc.value) + assert _failed_check_names(exc.value) == ["turn2_switched"] + assert client.send_message.call_count == 1 + client.list_items.assert_not_called() + + +def test_a_chat_error_stops_the_run(): + client = _client([ChatError("gen-ai fell over")], []) + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client) + assert _failed_check_names(exc.value) == ["turn1_answered"] + client.switch_agent.assert_not_called() + + +# --------------------------------------------------------------------------- # +# per-turn code checks +# --------------------------------------------------------------------------- # +def _one_turn(checks: dict[str, Any]) -> dict[str, Any]: + return _item({"agent_id": "a", "turns": [{"message": "m", "checks": checks}]}) + + +@pytest.mark.parametrize( + ("checks", "used", "failed"), + [ + ({"skills_used": ["alert"]}, ["alert", "metric"], []), + ({"skills_used": ["alert"]}, ["metric"], ["turn1_skills_used"]), + ({"skills_used": ["alert"]}, [], ["turn1_skills_used"]), + ({"skills_not_used": ["visualization"]}, ["alert"], []), + ({"skills_not_used": ["visualization"]}, ["visualization"], ["turn1_skills_not_used"]), + ], +) +def test_skills_checks_read_the_turn_s_tool_calls(checks, used, failed): + client = _client([_chat()], _items((None, "a", used))) + if not failed: + assert _evaluate(client, _one_turn(checks)).runs_passed == 1 + return + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client, _one_turn(checks)) + assert _failed_check_names(exc.value) == failed + + +def test_a_carried_over_skill_used_without_set_skills_passes_skills_used(): + spec = { + "agent_id": "planner", + "turns": [{"message": "a"}, {"message": "b", "switch_to": "builder", "checks": {"skills_used": ["alert"]}}], + } + # set_skills only in turn 1; turn 2 runs an alert tool with no set_skills call + client = _client( + [_chat(skills=["alert"]), _chat()], + _items((None, "planner", ["alert"]), ("builder", "builder", ["alert"])), + ) + outcome = _evaluate(client, _item(spec)) + assert outcome.runs_passed == 1 + assert [t["skills"] for t in outcome.detail["turns"]] == [["alert"], ["alert"]] + + +def test_a_requested_skill_whose_tools_never_ran_passes_skills_not_used(): + client = _client([_chat(skills=["key_driver_analysis"])], _items((None, "a", []))) + assert _evaluate(client, _one_turn({"skills_not_used": ["key_driver_analysis"]})).runs_passed == 1 + + +def test_skills_of_one_turn_do_not_leak_into_the_next(): + spec = {"agent_id": "a", "turns": [{"message": "a"}, {"message": "b", "checks": {"skills_not_used": ["alert"]}}]} + client = _client([_chat(), _chat()], _items((None, "a", ["alert"]), (None, "a", ["visualization"]))) + assert _evaluate(client, _item(spec)).runs_passed == 1 + + +def test_unreadable_items_fail_every_skills_check(): + spec = { + "agent_id": "a", + "turns": [ + {"message": "a", "checks": {"skills_used": ["alert"]}}, + {"message": "b", "checks": {"skills_not_used": ["visualization"]}}, + ], + } + client = _client([_chat(), _chat()], []) + request = httpx.Request("GET", "https://h/items") + client.list_items.side_effect = httpx.HTTPStatusError( + "500", request=request, response=httpx.Response(500, request=request) + ) + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client, _item(spec)) + checks = exc.value.failed_runs[0]["failed_checks"] + assert [c["name"] for c in checks] == ["items_read", "turn1_skills_used", "turn2_skills_not_used"] + assert checks[1]["actual"] == "items unreadable" + assert exc.value.failed_runs[0]["exit_reason"] == "chat_error" + + +def _proposal(*metric_ids: str) -> dict[str, Any]: + measures = [ + {"localIdentifier": f"m{i}", "definition": {"measure": {"item": {"identifier": {"id": m}}}}} + for i, m in enumerate(metric_ids, start=1) + ] + return {"cta": "Create?", "alert": {"execution": {"measures": measures}}} + + +@pytest.mark.parametrize( + ("expected", "proposals", "ok"), + [ + ("revenue", [_proposal("revenue")], True), + ("metric/revenue", [_proposal("revenue")], True), + ("revenue", [_proposal("orders")], False), + # revenue is only the comparison side + ("revenue", [_proposal("orders", "revenue")], False), + ("revenue", [], False), + ], +) +def test_alert_metric_reads_the_proposal_measure(expected, proposals, ok): + client = _client([_chat(proposals=proposals)], _items((None, "a"))) + item = _one_turn({"alert_metric": expected}) + if ok: + assert _evaluate(client, item).runs_passed == 1 + return + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client, item) + assert _failed_check_names(exc.value) == ["turn1_alert_metric"] + + +_VIZ = { + "type": "bar_chart", + "query": {"fields": {"m": {"using": "metric/revenue"}, "o": {"using": "metric/orders"}, "r": "label/region"}}, + "metrics": ["m", "o"], + "view_by": ["r"], +} + + +@pytest.mark.parametrize( + ("expected", "viz", "ok"), + [ + ({"metrics": ["revenue"], "attributes": ["region"]}, _VIZ, True), + ({"metrics": ["metric/revenue"], "attributes": ["label/region"]}, _VIZ, True), + ({"metrics": ["revenue"], "attributes": ["country"]}, _VIZ, False), + ({"metrics": ["revenue"]}, None, False), + ], +) +def test_visualization_is_a_subset_match(expected, viz, ok): + client = _client([_chat(viz=viz)], _items((None, "a"))) + item = _one_turn({"visualization": expected}) + if ok: + assert _evaluate(client, item).runs_passed == 1 + return + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client, item) + assert _failed_check_names(exc.value) == ["turn1_visualization"] + + +# --------------------------------------------------------------------------- # +# judge +# --------------------------------------------------------------------------- # +def _judged_spec() -> dict[str, Any]: + return { + "agent_id": "planner", + "turns": [ + {"message": "Plan: watch revenue by region weekly."}, + { + "message": "Now set it up.", + "switch_to": "builder", + "checks": {"must_include": ["Restates the plan"], "must_not_include": ["Asks for the plan again"]}, + }, + ], + } + + +def test_the_judge_sees_the_transcript_labelled_by_agent(): + client = _client( + [_chat("Here is the plan"), _chat("Setting up")], _items((None, "planner"), ("builder", "builder")) + ) + positive = _judge() + outcome = _evaluate(client, _item(_judged_spec()), positive) + score_calls = positive.score.call_args_list + + assert len(score_calls) == 1 + for c in score_calls: + text = c.kwargs["input"] + assert "USER: Plan: watch revenue by region weekly." in text + assert "ASSISTANT (agent planner): Here is the plan" in text + assert "LATEST MESSAGE: Now set it up." in text + assert "ANSWERED BY: agent builder" in text + assert c.kwargs["actual_output"] == "Setting up" + assert outcome.detail["turn2_include_0"] is True + # The violation judge found nothing, so the exclude criterion holds. + assert outcome.detail["turn2_exclude_0"] is True + + +def test_a_failed_criterion_is_named_with_the_judge_reason(): + client = _client([_chat(), _chat()], _items((None, "planner"), ("builder", "builder"))) + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client, _item(_judged_spec()), positive=_judge(verdict=False)) + checks = exc.value.failed_runs[0]["failed_checks"] + assert [c["name"] for c in checks] == ["turn2_include_0"] + assert checks[0]["part"] == "judge" + + +def test_no_judge_is_built_for_an_item_without_criteria(): + client = _client([_chat(), _chat()], _items((None, "planner"), ("builder", "builder"))) + with _patched(client) as mock_judge: + evaluate_agentic_agent_switcher("https://h", "tok", "ws1", _item()) + mock_judge.assert_not_called() + + +# --------------------------------------------------------------------------- # +# K runs +# --------------------------------------------------------------------------- # +def test_each_run_replays_on_a_new_conversation(): + answers = [_chat(), _chat(), _chat(), _chat()] + client = _client(answers, _items((None, "planner"), ("builder", "builder")), conversations=2) + outcome = _evaluate(client, k=2, gate="power") + + assert client.create_conversation.call_count == 2 + assert [c.args[0] for c in client.send_message.call_args_list] == ["conv-1", "conv-1", "conv-2", "conv-2"] + assert [c.args for c in client.switch_agent.call_args_list] == [("conv-1", "builder"), ("conv-2", "builder")] + assert [c.args[0] for c in client.delete_conversation.call_args_list] == ["conv-1", "conv-2"] + assert (outcome.runs_passed, outcome.runs_effective) == (2, 2) + + +def test_pass_power_k_fails_when_one_run_fails(): + good = _items((None, "planner"), ("builder", "builder")) + bad = _items((None, "planner"), ("builder", "planner")) + client = _client([_chat()] * 4, good, conversations=2) + client.list_items.side_effect = [good, bad] + with pytest.raises(AgentSwitcherAssertionError) as exc: + _evaluate(client, k=2, gate="power") + assert exc.value.runs_passed == 1 + assert [r["run_index"] for r in exc.value.failed_runs] == [2] + + +def test_k_below_one_is_refused(): + with pytest.raises(ValueError, match="k must be at least 1"): + run_agentic_agent_switcher("https://h", "tok", "ws1", AgentSwitcherInput.model_validate(_TWO_TURNS), k=0) + + +# --------------------------------------------------------------------------- # +# Langfuse scores +# --------------------------------------------------------------------------- # +class _ScoreCtx: + def __init__(self) -> None: + self.run_metadata: dict[str, Any] = {} + self.scores: list[tuple[int, str, Any]] = [] + + def trace(self, _conversation_id: str) -> None: + return None + + @contextlib.contextmanager + def observe(self, _trace: object, run_idx: int, **_kwargs: Any): + yield run_idx + + def score(self, trace_id: int, *, name: str, value: Any, data_type: str, comment: str | None = None) -> None: + self.scores.append((trace_id, name, value)) + + def quality(self, trace_id: int, **_kwargs: Any) -> None: + self.scores.append((trace_id, "quality_score", None)) + + +def test_each_run_is_scored_with_the_switch_checks(): + captured: dict[str, Any] = {} + + def _capture(_submit: object, _identity: object, **kwargs: Any) -> None: + captured.update(kwargs) + + spec = { + "agent_id": "planner", + "turns": [{"message": "a"}, {"message": "b", "switch_to": "builder", "checks": {"skills_used": ["alert"]}}], + } + client = _client([_chat(), _chat()], _items((None, "planner"), ("builder", "builder", ["alert"]))) + with patch(f"{_MODULE}.submit_trace_scoring", _capture): + _evaluate(client, _item(spec), langfuse=MagicMock(), gate="power") + ctx = _ScoreCtx() + captured["write_scores"](ctx) + + assert captured["conversation_ids"] == ["conv-1"] + scores = {name: value for _, name, value in ctx.scores} + assert scores["agent_switcher_pass"] == 1.0 + assert scores["agent_stamped"] == 1.0 + assert scores["switch_marker"] == 1.0 + assert scores["turn_checks_pass"] == 1.0 + assert ctx.run_metadata == {"eval_k": 1, "eval_gate": "power"} diff --git a/packages/gooddata-eval/tests/test_agentic_runner.py b/packages/gooddata-eval/tests/test_agentic_runner.py index 26abd44e2..696a3fc75 100644 --- a/packages/gooddata-eval/tests/test_agentic_runner.py +++ b/packages/gooddata-eval/tests/test_agentic_runner.py @@ -107,8 +107,12 @@ def test_dispatch_agentic_omits_agent_id_by_default(): ), ("agentic_anomaly_detection", {"metric": "metric/spend"}, "evaluate_agentic_anomaly_detection"), ("agentic_what_if", {"metric_id": "spend"}, "evaluate_agentic_what_if"), + ("agentic_agent_switcher", {}, "evaluate_agentic_agent_switcher"), ] +# Kinds whose item names its own agents, so the CLI's --agent-id does not reach them. +_KINDS_WITH_OWN_AGENT = {"agentic_agent_switcher"} + # The one kind whose dispatch needs more than question/expected_output: the dashboard to # summarize is named in summary_input, exactly as for the single-shot dashboard_summary. _SUMMARY_INPUT_KINDS = {"agentic_dashboard_summary"} @@ -144,6 +148,9 @@ def test_dispatch_agentic_passes_agent_id_through_for_every_kind(kind, expected_ model_version_override=None, agent_id="agent-1", ) + if kind in _KINDS_WITH_OWN_AGENT: + assert "agent_id" not in mock_eval.call_args.kwargs + return assert mock_eval.call_args.kwargs["agent_id"] == "agent-1" @@ -895,6 +902,7 @@ def test_an_errored_item_without_timings_keeps_its_zero_defaults(): ("agentic_forecasting", "evaluate_agentic_forecasting", {}), ("agentic_dashboard_summary", "evaluate_agentic_dashboard_summary", {}), ("agentic_conversation", "evaluate_agentic_conversation", {"id": "c1", "expected_skills": [], "turns": []}), + ("agentic_agent_switcher", "evaluate_agentic_agent_switcher", {}), ] @@ -938,7 +946,8 @@ def test_dispatch_agentic_passes_user_context_through_to_every_kind( # agentic_dashboard_summary builds its own context from the dashboard it summarises and # takes its scope from summary_input, so relaying the item's context would override the # thing under test rather than add to it. Excluded on purpose, not by omission. -_KINDS_WITHOUT_USER_CONTEXT = {"agentic_dashboard_summary"} +# agentic_agent_switcher gets the whole item and reads user_context off it. +_KINDS_WITHOUT_USER_CONTEXT = {"agentic_dashboard_summary", "agentic_agent_switcher"} class _Stop(Exception): diff --git a/packages/gooddata-eval/tests/test_sse_client.py b/packages/gooddata-eval/tests/test_sse_client.py index 6f28a767a..4e5afdf24 100644 --- a/packages/gooddata-eval/tests/test_sse_client.py +++ b/packages/gooddata-eval/tests/test_sse_client.py @@ -810,6 +810,56 @@ def handler(request): assert seen["body"] == {"agentId": "agent-123"} +def test_switch_agent_posts_the_agent_and_returns_the_conversation(): + seen = {} + + def handler(request): + seen["url"] = str(request.url) + seen["body"] = json.loads(request.content) + return httpx.Response(200, json={"conversationId": "c1", "agentId": "builder"}) + + client = _client_with_handler(handler) + assert client.switch_agent("c1", "builder") == {"conversationId": "c1", "agentId": "builder"} + assert seen["url"].endswith("/chat/conversations/c1/switchAgent") + assert seen["body"] == {"agentId": "builder"} + + +def test_switch_agent_retries_5xx_but_not_404(monkeypatch): + monkeypatch.setattr(sse_mod.time, "sleep", lambda s: None) + statuses = iter([503, 200]) + client = _client_with_handler(lambda request: httpx.Response(next(statuses), json={"agentId": "b"})) + assert client.switch_agent("c1", "b") == {"agentId": "b"} + + calls = {"n": 0} + + def not_found(request): + calls["n"] += 1 + return httpx.Response(404) + + with pytest.raises(httpx.HTTPStatusError): + _client_with_handler(not_found).switch_agent("c1", "b") + assert calls["n"] == 1 + + +def test_list_items_returns_the_items_list(): + items = [{"role": "user", "agentId": "a"}, {"role": "assistant", "agentId": "a"}] + seen = {} + + def handler(request): + seen["url"] = str(request.url) + return httpx.Response(200, json={"items": items}) + + client = _client_with_handler(handler) + assert client.list_items("c1") == items + assert seen["url"].endswith("/chat/conversations/c1/items") + + +def test_list_items_rejects_a_body_without_items(): + client = _client_with_handler(lambda request: httpx.Response(200, json={"data": []})) + with pytest.raises(ValueError, match="items"): + client.list_items("c1") + + def test_int_env_uses_default_when_unset(monkeypatch): monkeypatch.delenv("GD_TEST_INT", raising=False) assert sse_mod._int_env("GD_TEST_INT", 5) == 5 diff --git a/packages/gooddata-eval/tests/test_trace_linker.py b/packages/gooddata-eval/tests/test_trace_linker.py index cf60029f9..5bee29bd2 100644 --- a/packages/gooddata-eval/tests/test_trace_linker.py +++ b/packages/gooddata-eval/tests/test_trace_linker.py @@ -124,6 +124,7 @@ def test_run_trace_link_inline_runs_the_task_on_the_calling_thread(): ("dashboard_summary", "evaluate_agentic_dashboard_summary"), ("anomaly_detection", "evaluate_agentic_anomaly_detection"), ("what_if", "evaluate_agentic_what_if"), + ("agent_switcher", "evaluate_agentic_agent_switcher"), ]