Skip to content

[bot] Merge master/4176545c into rel/dev - #1963

Merged
yenkins-admin merged 2 commits into
rel/devfrom
snapshot-master-4176545c-to-rel/dev
Nov 18, 2026
Merged

yenkins-admin merged 2 commits into
rel/devfrom
snapshot-master-4176545c-to-rel/dev

Conversation

@yenkins-admin

Copy link
Copy Markdown
Contributor

🚀 Automated PR to perform merge from master into rel/dev with changes up to 4176545 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/44236421185).

myhoai and others added 2 commits November 18, 2026 11:06
QA-30213, phase 1: a failing run is labelled ERROR only when the test
harness failed (its simulated user or its judge), FAIL otherwise. The
rule is fail-closed, in core/outcome.py; chat_error, agent_timeout,
turn_incomplete and any unknown cause stay FAIL, since a customer hits
them too. A judge failure is an ERROR only when the judge's verdict was
the one still open: a document run that already failed a fixed check
stays a FAIL whatever the judge would have said.

The labels decide nothing: an ERROR run still counts as a run that did
not pass, so gates, pass rates and scores are unchanged. They appear in
the "Failing runs" part of the failure message, in failed_runs[]
(exit_reason, outcome, failed_checks), on the raised exception or the
returned result (harness_error when every run errored, and shadow, the
verdict a phase-2 candidate rule would give, recorded only for
QA-30296), in the gd-eval run JSON report, and in Langfuse as an extra
eval_error score. Shadow is recorded on passing items too: under pass@K
an item can pass with most of its runs errored. An errored run that
gets no verdict is attached to its experiment item whether its trace
joined or not, as every scored run is.

An ERROR run still lists its failed checks, because the simulated user
is also asked after an answer that was already final. An item whose
every run errored says so first, and says no run was scored only when
every judge failed. The gate note names errored runs; under pass^K an
item is still called unstable when a run the agent failed sits beside
a passing one. A run with no chart, metric or alert names only that,
not every check that reads False without it. KDA, anomaly, what-if,
general question and guardrail name their failed checks as well. The
failing runs are cut at 2000 characters, at a run boundary; every run
stays in failed_runs[], whose outcome is read from the same labels as
the message and the eval_error score.

Judge failures of general question, guardrail and document skill are
labelled too; a failed takeaway request stays FAIL. A timed-out or
incomplete turn records agent_timeout / turn_incomplete instead of
chat_error, including an agent that goes silent past its budget and
ends in a read timeout, whether mid-stream or awaiting the headers.

jira: QA-30213
risk: low
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(gooddata-eval): label harness errors, keep every verdict
@yenkins-admin
yenkins-admin merged commit abd78f0 into rel/dev Nov 18, 2026
1 check passed
@yenkins-admin
yenkins-admin deleted the snapshot-master-4176545c-to-rel/dev branch November 18, 2026 04:19
@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a9f3d6d1-00c1-439a-8bb2-b64e58920397

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@codecov

codecov Bot commented Nov 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.43590% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.91%. Comparing base (c249199) to head (4176545).
⚠️ Report is 655 commits behind head on rel/dev.

Files with missing lines Patch % Lines
...al/src/gooddata_eval/core/agentic/visualization.py 82.69% 9 Missing ⚠️
...a-eval/src/gooddata_eval/core/agentic/guardrail.py 87.50% 1 Missing ⚠️
...val/src/gooddata_eval/core/agentic/metric_skill.py 92.85% 1 Missing ⚠️
...ata-eval/src/gooddata_eval/core/chat/sse_client.py 96.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           rel/dev    #1963      +/-   ##
===========================================
+ Coverage    86.68%   86.91%   +0.23%     
===========================================
  Files          350      352       +2     
  Lines        27283    27694     +411     
===========================================
+ Hits         23649    24070     +421     
+ Misses        3634     3624      -10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants