Repository navigation
[bot] Merge master/243ec3ea into rel/dev - #1903
Merged
Merged
Conversation
The alert-skill evaluator never compared the period comparison, so an alert created for "decreases by more than 15% compared with the previous period" passed every check when the agent compared against the same period a year ago, went back two periods, compared absolute values instead of a percentage, or hand-built a "previous period" metric and passed it as compare_metric_id. expected_output gains an optional Comparison key, normalized onto CatalogMetricAlert.comparison and checked by comparison_correct, which joins strict_pass, the Langfuse scores and the run detail. It follows the three-state convention of filters and attributes: an object asserts the properties it states (kind, periods_ago, date_dataset_id, date_attribute_id, arithmetic_operator), "None" asserts the alert has no comparison at all, and an absent or empty value is not asserted, so existing items are unaffected. The actual arguments are read the way create_metric_alert resolves them: a comparison_kind left out next to compare_metric_id is METRIC, and on an alert that does compare, an omitted property counts as the tool's default: arithmetic_operator DIFFERENCE on any comparison, periods_ago 1 only on a period comparison (PREVIOUS_PERIOD, SAME_PERIOD_PRIOR_YEAR). A metric-to-metric comparison has no periods, so a hand-built "previous period" metric cannot satisfy a stated periods_ago, and neither can an alert with no comparison at all. The comparison period (DAY, MONTH, ...) is not part of the check: it is the granularity of the comparison date filter, which filters_correct already asserts. jira: QA-29243 risk: low Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The alert-skill evaluator never compared the alert's name or the dashboard it is bound to. An agent that dropped the requested title, so that create_metric_alert saved an auto-generated name and kept the requested one only as an automation_id slug, passed every check, and so did an alert bound to the wrong dashboard. expected_output gains two optional keys, each asserted only when the item states a value; absent or blank means unasserted, so existing items are unaffected: - Title: exact match against the title argument. The fixture value is trimmed, the agent's is not: a padded name is not the requested one. An omitted title fails even when the automation_id carries its slug. - Dashboard: "Title (id)", read the way Metric is, or a dashboard_id key; compared against the dashboard_id argument verbatim. A Dashboard with no "(id)" raises before the run, since a title alone can never match and would leave the item red on every run. title_correct and dashboard_correct join strict_pass, the Langfuse scores and the run detail only for an item that states them, and so does comparison_correct: an unstated check is None, neither scored nor gated. Otherwise the 55 items with no title would each log title_correct=1 for a check that never ran, and a refusal would read as a title, dashboard and comparison failure on items that ask for none. A stated title is also given to the simulated user, which then refuses a proposal showing another name, as a real user would. The dashboard is not: finding it is the behaviour under test, and telling the simulated user would hand the agent the answer. The per-check list that strict_pass, the best-run key, the run detail, the Langfuse scores and the assertion message each repeated is now AlertEvaluation.checks(), built from explicit check names, so a check added later reaches all five and a non-check field never does. Status is left out: create_metric_alert has no status argument and an alert is always created active. jira: QA-29244 risk: low Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rtions fix(gooddata-eval): assert alert comparison, title and dashboard
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1903 +/- ##
===========================================
+ Coverage 85.65% 85.71% +0.06%
===========================================
Files 343 343
Lines 25069 25163 +94
===========================================
+ Hits 21472 21569 +97
+ Misses 3597 3594 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to 243ec3e (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/40315108140).