Skip to content

Add opt-in @agent-pointer-eval mixed harness for consumer AI pointer tests - #11

Merged
ds-code-t merged 1 commit into
2.1.10from
cursor/agent-pointer-eval-b12b
Aug 27, 2026
Merged

Add opt-in @agent-pointer-eval mixed harness for consumer AI pointer tests#11
ds-code-t merged 1 commit into
2.1.10from
cursor/agent-pointer-eval-b12b

Conversation

@ds-code-t

@ds-code-t ds-code-t commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a canned mixed pass/fail Gherkin suite so Dan can score whether a consumer AI agent follows the short AGENTS.md pointer (export-guidance.pickleball/AGENT-GUIDE.md) and then uses discover-then-isolate, without coaching the eval tag in the pointer files.

Target is 2.1.10, not master.

Functionality-change coverage

  • I reviewed AGENTS.md and the relevant feature-map entry.
  • Framework implementation is complete. (No framework runtime change; packaged guidance snapshot updated.)
  • Focused framework tests were added or updated. (Contract check in scripts/verify_agent_contract.py.)
  • Maven consumer scenarios and supporting resources were added or updated when applicable.
  • README or canonical documentation was updated when behavior changed.
  • docs/agent/feature-map.md remains accurate.
  • docs/agent/repository-index.md is current.
  • Backward compatibility was preserved, or the breaking change is documented.

What landed

  • New feature: maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature
  • Tagged only @agent-pointer-eval (plus optional @agent-pointer-eval-pass / @agent-pointer-eval-fail)
  • Not tagged @all, @regression, @browser, or any other Maven suite-profile tag
  • Short maintainer note in docs/consumer-agent-guide.md (packaged AGENT-GUIDE.md kept in sync)
  • Consumer AGENTS.md / Copilot pointer files are unchanged
  • Contract: feature exists, Feature/Scenario tags stay on @agent-pointer-eval*, pointer files must not mention the tag

How to run

./maven-consumer-project/mvnw -f maven-consumer-project/pom.xml test \
  -Dpkb_runvars="pkb_tags=@agent-pointer-eval, pkb_browser=CHROME_HEADLESS"

Expected mixed results (do not treat as product failures):

Scenario Expected
Agent pointer eval passing pickleball equality PASS
Agent pointer eval passing pointer token equality PASS
Agent pointer eval failing fruit mismatch FAIL (apple vs orange)
Agent pointer eval failing greek-letter mismatch FAIL (alpha vs omega)
Agent pointer eval passing home heading PASS
Agent pointer eval failing forms heading FAIL (wrong heading on URL.forms)

Observed on this branch with CHROME_HEADLESS: 3 passed, 3 failed, names as above.

Validation

  • python3 scripts/verify_agent_contract.py
  • python3 scripts/refresh_agent_index.py --check
  • python3 scripts/sync_consumer_guidance.py --check
  • ./gradlew test — BUILD SUCCESSFUL
  • ./gradlew publishToMavenLocal — BUILD SUCCESSFUL
  • Focused @agent-pointer-eval run with CHROME_HEADLESS — mixed 3 pass / 3 fail as designed
  • pkb_tags="@all and @agent-pointer-eval" — 0 selected, BUILD SUCCESS
  • pkb_tags="@regression and @agent-pointer-eval" — 0 selected, BUILD SUCCESS
  • pkb_tags="@browser and @agent-pointer-eval" — 0 selected, BUILD SUCCESS
  • pkb_tags=@diagnostic-reporting — PickleballGuidanceChecks passed, including packaged-guide assertions

Full @all was not used as a migration suite. Intersection tags above confirm these scenarios are outside default @all / @regression / @browser selection.

Open in Web Open in Cursor 

…tests.

Keep the canned pass/fail scenarios out of @ALL, @regression, and Maven suite
profiles so default consumer runs stay green. Document how maintainers launch
the harness without putting the tag in the short AGENTS.md pointer.

Co-authored-by: ds-code-t <ds-code-t@users.noreply.github.com>
@ds-code-t
ds-code-t marked this pull request as ready for review August 27, 2026 15:34
@ds-code-t
ds-code-t merged commit 71ae370 into 2.1.10 Aug 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants