Repository navigation
[bot] Merge master/06bf7727 into rel/dev - #1931
Merged
Merged
Conversation
`document_asked_first` was recorded but never scored, and the simulated user always answered with the expected charts and period. An item could not require the copilot to ask, and a question about something else got an answer that did not fit. Two optional expectation keys close that: - `asked_first_required`: true makes `document_asked_first` a strict check. It fails a run that drafted without asking on an earlier turn. A run that never drafted fails only for the missing draft. The key needs `expects_clarification: true`. The diagnostic copy is then not published, so the score is not reported twice. - `clarification_reply`: the text the simulated user sends to the first turn that asks, in place of the reply built from the charts and period. Later questions get the built reply. The key needs `expects_clarification: true`. A turn counts as asking when `classify_reply` returns anything but `no_action`. A confirmation such as "Should I create the document anyway?" stops the copilot before it drafts, so it counts too. The diagnostic follows the same rule. `classify_reply` itself is unchanged, so metric and alert items keep their replies. A malformed value is rejected before the first request. Without either key, nothing changes for existing items. A refused `draft_document` call is not a draft, so a copilot that asks after the tool refuses an empty page counts as asking first. jira: LX-3243 risk: low Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(gooddata-eval): gate a document item on asking before it drafts
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1931 +/- ##
===========================================
+ Coverage 86.15% 86.17% +0.01%
===========================================
Files 344 344
Lines 25874 25901 +27
===========================================
+ Hits 22291 22319 +28
+ Misses 3583 3582 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to 06bf772 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/42617264461).