Repository navigation
[bot] Merge master/4176545c into rel/dev - #1963
Merged
Merged
Conversation
QA-30213, phase 1: a failing run is labelled ERROR only when the test harness failed (its simulated user or its judge), FAIL otherwise. The rule is fail-closed, in core/outcome.py; chat_error, agent_timeout, turn_incomplete and any unknown cause stay FAIL, since a customer hits them too. A judge failure is an ERROR only when the judge's verdict was the one still open: a document run that already failed a fixed check stays a FAIL whatever the judge would have said. The labels decide nothing: an ERROR run still counts as a run that did not pass, so gates, pass rates and scores are unchanged. They appear in the "Failing runs" part of the failure message, in failed_runs[] (exit_reason, outcome, failed_checks), on the raised exception or the returned result (harness_error when every run errored, and shadow, the verdict a phase-2 candidate rule would give, recorded only for QA-30296), in the gd-eval run JSON report, and in Langfuse as an extra eval_error score. Shadow is recorded on passing items too: under pass@K an item can pass with most of its runs errored. An errored run that gets no verdict is attached to its experiment item whether its trace joined or not, as every scored run is. An ERROR run still lists its failed checks, because the simulated user is also asked after an answer that was already final. An item whose every run errored says so first, and says no run was scored only when every judge failed. The gate note names errored runs; under pass^K an item is still called unstable when a run the agent failed sits beside a passing one. A run with no chart, metric or alert names only that, not every check that reads False without it. KDA, anomaly, what-if, general question and guardrail name their failed checks as well. The failing runs are cut at 2000 characters, at a run boundary; every run stays in failed_runs[], whose outcome is read from the same labels as the message and the eval_error score. Judge failures of general question, guardrail and document skill are labelled too; a failed takeaway request stays FAIL. A timed-out or incomplete turn records agent_timeout / turn_incomplete instead of chat_error, including an agent that goes silent past its budget and ends in a read timeout, whether mid-stream or awaiting the headers. jira: QA-30213 risk: low Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(gooddata-eval): label harness errors, keep every verdict
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1963 +/- ##
===========================================
+ Coverage 86.68% 86.91% +0.23%
===========================================
Files 350 352 +2
Lines 27283 27694 +411
===========================================
+ Hits 23649 24070 +421
+ Misses 3634 3624 -10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to 4176545 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/44236421185).