Add opt-in @agent-pointer-eval mixed harness for consumer AI pointer tests - #11
Merged
Merged
Conversation
…tests. Keep the canned pass/fail scenarios out of @ALL, @regression, and Maven suite profiles so default consumer runs stay green. Document how maintainers launch the harness without putting the tag in the short AGENTS.md pointer. Co-authored-by: ds-code-t <ds-code-t@users.noreply.github.com>
ds-code-t
marked this pull request as ready for review
August 27, 2026 15:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a canned mixed pass/fail Gherkin suite so Dan can score whether a consumer AI agent follows the short
AGENTS.mdpointer (export-guidance→.pickleball/AGENT-GUIDE.md) and then uses discover-then-isolate, without coaching the eval tag in the pointer files.Target is 2.1.10, not master.
Functionality-change coverage
AGENTS.mdand the relevant feature-map entry.scripts/verify_agent_contract.py.)docs/agent/feature-map.mdremains accurate.docs/agent/repository-index.mdis current.What landed
maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature@agent-pointer-eval(plus optional@agent-pointer-eval-pass/@agent-pointer-eval-fail)@all,@regression,@browser, or any other Maven suite-profile tagdocs/consumer-agent-guide.md(packagedAGENT-GUIDE.mdkept in sync)AGENTS.md/ Copilot pointer files are unchanged@agent-pointer-eval*, pointer files must not mention the tagHow to run
Expected mixed results (do not treat as product failures):
applevsorange)alphavsomega)URL.forms)Observed on this branch with
CHROME_HEADLESS: 3 passed, 3 failed, names as above.Validation
python3 scripts/verify_agent_contract.pypython3 scripts/refresh_agent_index.py --checkpython3 scripts/sync_consumer_guidance.py --check./gradlew test— BUILD SUCCESSFUL./gradlew publishToMavenLocal— BUILD SUCCESSFUL@agent-pointer-evalrun withCHROME_HEADLESS— mixed 3 pass / 3 fail as designedpkb_tags="@all and @agent-pointer-eval"— 0 selected, BUILD SUCCESSpkb_tags="@regression and @agent-pointer-eval"— 0 selected, BUILD SUCCESSpkb_tags="@browser and @agent-pointer-eval"— 0 selected, BUILD SUCCESSpkb_tags=@diagnostic-reporting— PickleballGuidanceChecks passed, including packaged-guide assertionsFull
@allwas not used as a migration suite. Intersection tags above confirm these scenarios are outside default@all/@regression/@browserselection.