Skip to content

[bot] Merge master/243ec3ea into rel/dev - #1903

Merged
yenkins-admin merged 3 commits into
rel/devfrom
snapshot-master-243ec3ea-to-rel/dev
Oct 26, 2026
Merged

yenkins-admin merged 3 commits into
rel/devfrom
snapshot-master-243ec3ea-to-rel/dev

Conversation

@yenkins-admin

Copy link
Copy Markdown
Contributor

🚀 Automated PR to perform merge from master into rel/dev with changes up to 243ec3e (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/40315108140).

myhoai and others added 3 commits October 26, 2026 17:05
The alert-skill evaluator never compared the period comparison, so an
alert created for "decreases by more than 15% compared with the previous
period" passed every check when the agent compared against the same
period a year ago, went back two periods, compared absolute values
instead of a percentage, or hand-built a "previous period" metric and
passed it as compare_metric_id.

expected_output gains an optional Comparison key, normalized onto
CatalogMetricAlert.comparison and checked by comparison_correct, which
joins strict_pass, the Langfuse scores and the run detail. It follows
the three-state convention of filters and attributes: an object asserts
the properties it states (kind, periods_ago, date_dataset_id,
date_attribute_id, arithmetic_operator), "None" asserts the alert has no
comparison at all, and an absent or empty value is not asserted, so
existing items are unaffected.

The actual arguments are read the way create_metric_alert resolves them:
a comparison_kind left out next to compare_metric_id is METRIC, and on
an alert that does compare, an omitted property counts as the tool's
default: arithmetic_operator DIFFERENCE on any comparison, periods_ago 1
only on a period comparison (PREVIOUS_PERIOD, SAME_PERIOD_PRIOR_YEAR).
A metric-to-metric comparison has no periods, so a hand-built "previous
period" metric cannot satisfy a stated periods_ago, and neither can an
alert with no comparison at all.

The comparison period (DAY, MONTH, ...) is not part of the check: it is
the granularity of the comparison date filter, which filters_correct
already asserts.

jira: QA-29243
risk: low
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The alert-skill evaluator never compared the alert's name or the
dashboard it is bound to. An agent that dropped the requested title, so
that create_metric_alert saved an auto-generated name and kept the
requested one only as an automation_id slug, passed every check, and so
did an alert bound to the wrong dashboard.

expected_output gains two optional keys, each asserted only when the
item states a value; absent or blank means unasserted, so existing items
are unaffected:

- Title: exact match against the title argument. The fixture value is
  trimmed, the agent's is not: a padded name is not the requested one. An
  omitted title fails even when the automation_id carries its slug.
- Dashboard: "Title (id)", read the way Metric is, or a dashboard_id
  key; compared against the dashboard_id argument verbatim. A Dashboard
  with no "(id)" raises before the run, since a title alone can never
  match and would leave the item red on every run.

title_correct and dashboard_correct join strict_pass, the Langfuse
scores and the run detail only for an item that states them, and so
does comparison_correct: an unstated check is None, neither scored nor
gated. Otherwise the 55 items with no title would each log
title_correct=1 for a check that never ran, and a refusal would read as
a title, dashboard and comparison failure on items that ask for none. A stated title is also given to the
simulated user, which then refuses a proposal showing another name, as
a real user would. The dashboard is not: finding it is the behaviour
under test, and telling the simulated user would hand the agent the
answer.

The per-check list that strict_pass, the best-run key, the run detail,
the Langfuse scores and the assertion message each repeated is now
AlertEvaluation.checks(), built from explicit check names, so a check
added later reaches all five and a non-check field never does.

Status is left out: create_metric_alert has no status argument and an
alert is always created active.

jira: QA-29244
risk: low
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rtions

fix(gooddata-eval): assert alert comparison, title and dashboard
@yenkins-admin
yenkins-admin merged commit 3566fb0 into rel/dev Oct 26, 2026
1 check passed
@yenkins-admin
yenkins-admin deleted the snapshot-master-243ec3ea-to-rel/dev branch October 26, 2026 11:27
@coderabbitai

coderabbitai Bot commented Oct 26, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ceb761cd-34a2-49b6-912c-26cbbf069883

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.71%. Comparing base (71c9de7) to head (243ec3e).
⚠️ Report is 628 commits behind head on rel/dev.

Additional details and impacted files
@@             Coverage Diff             @@
##           rel/dev    #1903      +/-   ##
===========================================
+ Coverage    85.65%   85.71%   +0.06%     
===========================================
  Files          343      343              
  Lines        25069    25163      +94     
===========================================
+ Hits         21472    21569      +97     
+ Misses        3597     3594       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants