Skip to content

[bot] Merge master/06bf7727 into rel/dev - #1931

Merged
yenkins-admin merged 2 commits into
rel/devfrom
snapshot-master-06bf7727-to-rel/dev
Nov 9, 2026
Merged

yenkins-admin merged 2 commits into
rel/devfrom
snapshot-master-06bf7727-to-rel/dev

Conversation

@yenkins-admin

Copy link
Copy Markdown
Contributor

🚀 Automated PR to perform merge from master into rel/dev with changes up to 06bf772 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/42617264461).

Roman Rakus and others added 2 commits November 9, 2026 11:41
`document_asked_first` was recorded but never scored, and the simulated
user always answered with the expected charts and period. An item could
not require the copilot to ask, and a question about something else got
an answer that did not fit.

Two optional expectation keys close that:

- `asked_first_required`: true makes `document_asked_first` a strict
  check. It fails a run that drafted without asking on an earlier turn.
  A run that never drafted fails only for the missing draft. The key
  needs `expects_clarification: true`. The diagnostic copy is then not
  published, so the score is not reported twice.
- `clarification_reply`: the text the simulated user sends to the first
  turn that asks, in place of the reply built from the charts and
  period. Later questions get the built reply. The key needs
  `expects_clarification: true`.

A turn counts as asking when `classify_reply` returns anything but
`no_action`. A confirmation such as "Should I create the document
anyway?" stops the copilot before it drafts, so it counts too. The
diagnostic follows the same rule. `classify_reply` itself is unchanged,
so metric and alert items keep their replies.

A malformed value is rejected before the first request. Without either
key, nothing changes for existing items. A refused `draft_document` call
is not a draft, so a copilot that asks after the tool refuses an empty
page counts as asking first.

jira: LX-3243
risk: low
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(gooddata-eval): gate a document item on asking before it drafts
@yenkins-admin
yenkins-admin merged commit 052d9a9 into rel/dev Nov 9, 2026
1 check passed
@yenkins-admin
yenkins-admin deleted the snapshot-master-06bf7727-to-rel/dev branch November 9, 2026 12:28
@codecov

codecov Bot commented Nov 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.17%. Comparing base (c226287) to head (06bf772).
⚠️ Report is 642 commits behind head on rel/dev.

Additional details and impacted files
@@             Coverage Diff             @@
##           rel/dev    #1931      +/-   ##
===========================================
+ Coverage    86.15%   86.17%   +0.01%     
===========================================
  Files          344      344              
  Lines        25874    25901      +27     
===========================================
+ Hits         22291    22319      +28     
+ Misses        3583     3582       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants