diff --git a/docs/agent/feature-map.md b/docs/agent/feature-map.md index 190a1f63..419166f9 100644 --- a/docs/agent/feature-map.md +++ b/docs/agent/feature-map.md @@ -20,6 +20,7 @@ This file maps consumer-visible capabilities to implementation anchors, executab | Mapping, ParsingMap/NodeMap, templates/directives | `MappingSteps.java`; `FileAndDataParsing.java`; `MappingProcessor.java`; `QuoteParser.java`; `NodeMap.java`; `ParsingMap.java`; `ValueFormatting.java`; `common/dataelements` | `mapping-and-resources.feature`; `mapping-value-type-preservation.feature`; `scenario-data-references.feature`; Data Element features; `QuoteParserChecks.java`; internal Java checks | `docs/mapping-and-templating.md`; `docs/data-values-and-elements.md`; `docs/data-element-query-runtime.md`; `docs/config-files-and-resource-mapping.md` | | Configuration/profiles/RunVars | `PKB_props.java`; `PickleballProfiles.java`; `PkbPropertyValueNormalizer.java`; runner/config classes; search `pkb_profile`, `pkb_runvars`, `pkb_run_profile`, `pkb_configpath` | `configuration-system-properties.feature`; `ProfileConfigurationChecks.java`; consumer properties/profile examples | `docs/configuration.md`; `docs/getting-started.md`; `docs/ai-run-configuration.md`; `docs/consumer-project.md` | | Consumer guidance export/reference snapshot | `DiagnosticCli.java`; `gradle/consumer-guidance.gradle`; `scripts/sync_consumer_guidance.py`; `maven-consumer-project/AGENTS.md`; `maven-consumer-project/.github/copilot-instructions.md`; search `export-guidance`, `discover-hint`, `GUIDANCE-MANIFEST.json`, `.pickleball/investigations` | `PickleballGuidanceChecks.java`; consumer guidance contract checks | `docs/consumer-agent-guide.md`; `docs/consumer-project.md` | +| Consumer AI-agent pointer-eval harness | `maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`; tag `@agent-pointer-eval` (optional `@agent-pointer-eval-pass` / `@agent-pointer-eval-fail`) | opt-in mixed pass/fail; must not carry `@all`, `@regression`, or other Maven suite-profile tags; `scripts/verify_agent_contract.py` | `docs/consumer-agent-guide.md` | | Diagnostic reporting and controlled reruns | `src/main/java/tools/dscode/common/reporting/diagnostic`; `InvestigationHandoff`; diagnostic aspects; `DiagnosticCli.java`; `emit-investigation`; `VisualFingerprintComparator.java`; `DiagnosticRunComparator.java` | `DiagnosticReportingChecks.java`; `Diagnostic213CompletionChecks.java`; `InvestigationHandoffChecks.java`; diagnostic features | `docs/diagnostic-reporting.md`; `docs/ai-diagnostic-reporting-plan.md`; `docs/ai-run-configuration.md`; root `AGENTS.md` | | Nested steps/block conditionals | search `Nested`, `Conditional`, `Block`, `Condition` in core implementation | `nested-and-block-conditionals.feature` | `docs/nested-steps.md`; `docs/block-conditionals.md` | | Component scenarios/reusable RUN/selectors/markers | `ModularScenarios.java`; `ScenarioStep.java`; `ScenarioStepData.java`; `StepBase.java`; `StepExtension.java`; `CurrentScenarioState.java`; `CucumberScanUtil.java`; search `finalizerSteps`, `RunSelection` | `component-scenarios.feature`; `reusable-scenario-selection.feature`; `run-step-parameter-variations.feature`; marker features | `docs/component-scenarios.md`; `docs/service-call-scenarios.md`; `docs/data-values-and-elements.md` | diff --git a/docs/agent/repository-index.md b/docs/agent/repository-index.md index 77f2a7b8..400252cc 100644 --- a/docs/agent/repository-index.md +++ b/docs/agent/repository-index.md @@ -604,6 +604,7 @@ This inventory helps coding agents discover relevant files. It does not replace ## Maven consumer feature scenarios +- `maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature` - `maven-consumer-project/src/test/resources/features/browser-action-contracts.feature` - `maven-consumer-project/src/test/resources/features/catalog-context.feature` - `maven-consumer-project/src/test/resources/features/component-scenarios.feature` diff --git a/docs/consumer-agent-guide.md b/docs/consumer-agent-guide.md index 18b9d23c..ea29adbc 100644 --- a/docs/consumer-agent-guide.md +++ b/docs/consumer-agent-guide.md @@ -295,3 +295,15 @@ Use `docs/consumer-project.md` on demand for the Maven consumer layout, local te If the consumer is nested inside the Pickleball source repository, repository-level `AGENTS.md` may impose additional maintainer rules for framework changes. Those core-maintainer rules are additive and do not replace this consumer-facing contract. For a normal external consumer, do not assume those core files exist. + +## Maintainer pointer-eval harness + +Pickleball's example Maven consumer includes an opt-in mixed pass/fail suite tagged only `@agent-pointer-eval` (`maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`). It is not part of `@all`, `@regression`, or the other Maven suite-profile tags. The failures are intentional canned fixtures for scoring whether a consumer AI agent follows the short `AGENTS.md` pointer into this guide and then uses discover-then-isolate. Do not treat those failures as product bugs, and do not "fix" the feature unless a human asked to change the harness. + +Run it explicitly: + +```text +mvn test -Dpkb_runvars="pkb_tags=@agent-pointer-eval, pkb_browser=CHROME_HEADLESS" +``` + +or `-Dpkb_tags=@agent-pointer-eval`. Do not add this tag to consumer `AGENTS.md` or Copilot pointer files. diff --git a/maven-consumer-project/src/test/java/tools/dscode/common/reporting/diagnostic/PickleballGuidanceChecks.java b/maven-consumer-project/src/test/java/tools/dscode/common/reporting/diagnostic/PickleballGuidanceChecks.java index eca48101..345b86e9 100644 --- a/maven-consumer-project/src/test/java/tools/dscode/common/reporting/diagnostic/PickleballGuidanceChecks.java +++ b/maven-consumer-project/src/test/java/tools/dscode/common/reporting/diagnostic/PickleballGuidanceChecks.java @@ -79,6 +79,8 @@ void dependencyPrintsCanonicalAgentGuide() { assertTrue(chooserList.contains("run-catalog.json")); assertTrue(chooserList.contains("after discovery has named the trouble spots")); assertTrue(chooserList.contains("Do not start a worker just to run the whole suite")); + assertTrue(guide.contains("Maintainer pointer-eval harness")); + assertTrue(guide.contains("@agent-pointer-eval")); String liveLoop = guide.substring( guide.indexOf("### Live isolation loop"), guide.indexOf("Generated guidance lifecycle") @@ -200,6 +202,9 @@ void dependencyExportsVersionMatchedGuidanceAndManifest() throws Exception { assertTrue(Files.isRegularFile(root.resolve( "maven-consumer-project/src/test/resources/features/dynamic-steps.feature" ))); + assertTrue(Files.isRegularFile(root.resolve( + "maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature" + ))); assertTrue(Files.isRegularFile(root.resolve( "maven-consumer-project/src/test/resources/calls/service-call-definitions.feature" ))); @@ -237,6 +242,9 @@ void dependencyExportsVersionMatchedGuidanceAndManifest() throws Exception { assertTrue(managedFiles.contains( "maven-consumer-project/src/test/resources/features/dynamic-steps.feature" )); + assertTrue(managedFiles.contains( + "maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature" + )); assertTrue(managedFiles.stream().noneMatch(path -> path.contains("_local2"))); assertFalse(managedFiles.contains("maven-consumer-project/AGENTS.md")); assertFalse(managedFiles.contains("maven-consumer-project/.github/copilot-instructions.md")); @@ -260,6 +268,8 @@ void dependencyExportsVersionMatchedGuidanceAndManifest() throws Exception { assertTrue(guide.contains("tell the human")); assertTrue(guide.contains("CLI/Maven-primary")); assertTrue(guide.contains("PickleballWorkbenchLauncher")); + assertTrue(guide.contains("Maintainer pointer-eval harness")); + assertTrue(guide.contains("@agent-pointer-eval")); String consumerProject = Files.readString(root.resolve("docs/consumer-project.md")); assertTrue(consumerProject.contains("keep console verbosity low")); diff --git a/maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature b/maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature new file mode 100644 index 00000000..0fe2fab0 --- /dev/null +++ b/maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature @@ -0,0 +1,32 @@ +@agent-pointer-eval +Feature: Agent pointer-eval mixed harness + Intentional mixed pass/fail harness for consumer AI-agent pointer tests. + It is not part of @all or @regression. Run it only with pkb_tags=@agent-pointer-eval + (or the pkb_runvars equivalent). Failures are canned and stable. Do not "fix" + this file as if it were product coverage unless the human asked to change the harness. + + @agent-pointer-eval @agent-pointer-eval-pass + Scenario: Agent pointer eval passing pickleball equality + * , verify "pickleball" equals "pickleball" + + @agent-pointer-eval @agent-pointer-eval-pass + Scenario: Agent pointer eval passing pointer token equality + * , verify "pointer-eval" equals "pointer-eval" + + @agent-pointer-eval @agent-pointer-eval-fail + Scenario: Agent pointer eval failing fruit mismatch + * , verify "apple" equals "orange" + + @agent-pointer-eval @agent-pointer-eval-fail + Scenario: Agent pointer eval failing greek-letter mismatch + * , verify "alpha" equals "omega" + + @agent-pointer-eval @agent-pointer-eval-pass + Scenario: Agent pointer eval passing home heading + * navigate to: URL.home + * , ensure "Pickleball Test Lab" Text is displayed + + @agent-pointer-eval @agent-pointer-eval-fail + Scenario: Agent pointer eval failing forms heading + * navigate to: URL.forms + * , ensure "AGENT POINTER EVAL INTENTIONAL FORMS HEADING" Text is displayed diff --git a/scripts/verify_agent_contract.py b/scripts/verify_agent_contract.py index 4fb41877..cd933bb1 100644 --- a/scripts/verify_agent_contract.py +++ b/scripts/verify_agent_contract.py @@ -346,6 +346,7 @@ def validate_consumer_bridge(errors: list[str]) -> None: "Diagnostic investigation protocol", "attach.json", "ui .", + "@agent-pointer-eval", ): if forbidden in text: errors.append( @@ -400,6 +401,7 @@ def validate_dependency_owned_guidance(errors: list[str]) -> None: "version-matched", "Generated Maven consumer reference", "maven-consumer-project/", + "@agent-pointer-eval", ): if required not in text: errors.append( @@ -449,6 +451,98 @@ def validate_consumer_ignore(errors: list[str]) -> None: ) +AGENT_POINTER_EVAL_FEATURE = ( + "maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature" +) +AGENT_POINTER_EVAL_TAG = "@agent-pointer-eval" +MAVEN_SUITE_PROFILE_TAGS = { + "@all", + "@regression", + "@smoke", + "@browser", + "@data", + "@navigation", + "@forms", + "@catalog", + "@mapping", + "@resources", + "@workflow", + "@block-conditionals", + "@nested-steps", + "@keyboard", + "@dialogs", + "@components", +} + + +def parse_gherkin_tagged_elements(text: str) -> list[tuple[str, list[str]]]: + pending: list[str] = [] + elements: list[tuple[str, list[str]]] = [] + for raw in text.splitlines(): + stripped = raw.strip() + if not stripped or stripped.startswith("#"): + continue + if stripped.startswith("@"): + pending.extend(token for token in stripped.split() if token.startswith("@")) + continue + if ( + stripped.startswith("Feature:") + or stripped.startswith("Scenario:") + or stripped.startswith("Scenario Outline:") + ): + elements.append((stripped, pending)) + pending = [] + return elements + + +def validate_agent_pointer_eval_harness(errors: list[str]) -> None: + relative = AGENT_POINTER_EVAL_FEATURE + path = ROOT / relative + if not path.is_file(): + errors.append("Missing opt-in consumer pointer-eval harness: " + relative) + return + + elements = parse_gherkin_tagged_elements(path.read_text(encoding="utf-8")) + if not elements or not elements[0][0].startswith("Feature:"): + errors.append("Pointer-eval harness must declare a Feature: " + relative) + return + + scenarios = [ + element + for element in elements + if element[0].startswith("Scenario:") or element[0].startswith("Scenario Outline:") + ] + if len(scenarios) < 2: + errors.append( + "Pointer-eval harness must contain more than one Scenario so isolate can " + "target distinct names: " + relative + ) + + for name, tags in elements: + missing = AGENT_POINTER_EVAL_TAG not in tags + forbidden = sorted(tag for tag in tags if tag in MAVEN_SUITE_PROFILE_TAGS) + extras = [ + tag + for tag in tags + if tag != AGENT_POINTER_EVAL_TAG and not tag.startswith(AGENT_POINTER_EVAL_TAG + "-") + ] + if missing: + errors.append( + f"Pointer-eval harness element must carry {AGENT_POINTER_EVAL_TAG}: " + f"{relative} -> {name}" + ) + if forbidden: + errors.append( + "Pointer-eval harness must not carry Maven suite-profile tags " + f"{forbidden}: {relative} -> {name}" + ) + if extras: + errors.append( + "Pointer-eval harness may only add unique @agent-pointer-eval-* tags, " + f"not {extras}: {relative} -> {name}" + ) + + def validate_packaged_guidance(errors: list[str]) -> None: script = ROOT / "scripts" / "sync_consumer_guidance.py" if not script.is_file(): @@ -542,6 +636,7 @@ def main() -> int: validate_consumer_bridge(errors) validate_consumer_readme(errors) validate_dependency_owned_guidance(errors) + validate_agent_pointer_eval_harness(errors) validate_consumer_reference_build_hook(errors) validate_consumer_tracked_artifacts(errors) validate_consumer_ignore(errors) diff --git a/src/main/resources/META-INF/pickleball/guidance/AGENT-GUIDE.md b/src/main/resources/META-INF/pickleball/guidance/AGENT-GUIDE.md index 18b9d23c..ea29adbc 100644 --- a/src/main/resources/META-INF/pickleball/guidance/AGENT-GUIDE.md +++ b/src/main/resources/META-INF/pickleball/guidance/AGENT-GUIDE.md @@ -295,3 +295,15 @@ Use `docs/consumer-project.md` on demand for the Maven consumer layout, local te If the consumer is nested inside the Pickleball source repository, repository-level `AGENTS.md` may impose additional maintainer rules for framework changes. Those core-maintainer rules are additive and do not replace this consumer-facing contract. For a normal external consumer, do not assume those core files exist. + +## Maintainer pointer-eval harness + +Pickleball's example Maven consumer includes an opt-in mixed pass/fail suite tagged only `@agent-pointer-eval` (`maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`). It is not part of `@all`, `@regression`, or the other Maven suite-profile tags. The failures are intentional canned fixtures for scoring whether a consumer AI agent follows the short `AGENTS.md` pointer into this guide and then uses discover-then-isolate. Do not treat those failures as product bugs, and do not "fix" the feature unless a human asked to change the harness. + +Run it explicitly: + +```text +mvn test -Dpkb_runvars="pkb_tags=@agent-pointer-eval, pkb_browser=CHROME_HEADLESS" +``` + +or `-Dpkb_tags=@agent-pointer-eval`. Do not add this tag to consumer `AGENTS.md` or Copilot pointer files. diff --git a/src/main/resources/META-INF/pickleball/guidance/docs/agent/feature-map.md b/src/main/resources/META-INF/pickleball/guidance/docs/agent/feature-map.md index 190a1f63..419166f9 100644 --- a/src/main/resources/META-INF/pickleball/guidance/docs/agent/feature-map.md +++ b/src/main/resources/META-INF/pickleball/guidance/docs/agent/feature-map.md @@ -20,6 +20,7 @@ This file maps consumer-visible capabilities to implementation anchors, executab | Mapping, ParsingMap/NodeMap, templates/directives | `MappingSteps.java`; `FileAndDataParsing.java`; `MappingProcessor.java`; `QuoteParser.java`; `NodeMap.java`; `ParsingMap.java`; `ValueFormatting.java`; `common/dataelements` | `mapping-and-resources.feature`; `mapping-value-type-preservation.feature`; `scenario-data-references.feature`; Data Element features; `QuoteParserChecks.java`; internal Java checks | `docs/mapping-and-templating.md`; `docs/data-values-and-elements.md`; `docs/data-element-query-runtime.md`; `docs/config-files-and-resource-mapping.md` | | Configuration/profiles/RunVars | `PKB_props.java`; `PickleballProfiles.java`; `PkbPropertyValueNormalizer.java`; runner/config classes; search `pkb_profile`, `pkb_runvars`, `pkb_run_profile`, `pkb_configpath` | `configuration-system-properties.feature`; `ProfileConfigurationChecks.java`; consumer properties/profile examples | `docs/configuration.md`; `docs/getting-started.md`; `docs/ai-run-configuration.md`; `docs/consumer-project.md` | | Consumer guidance export/reference snapshot | `DiagnosticCli.java`; `gradle/consumer-guidance.gradle`; `scripts/sync_consumer_guidance.py`; `maven-consumer-project/AGENTS.md`; `maven-consumer-project/.github/copilot-instructions.md`; search `export-guidance`, `discover-hint`, `GUIDANCE-MANIFEST.json`, `.pickleball/investigations` | `PickleballGuidanceChecks.java`; consumer guidance contract checks | `docs/consumer-agent-guide.md`; `docs/consumer-project.md` | +| Consumer AI-agent pointer-eval harness | `maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`; tag `@agent-pointer-eval` (optional `@agent-pointer-eval-pass` / `@agent-pointer-eval-fail`) | opt-in mixed pass/fail; must not carry `@all`, `@regression`, or other Maven suite-profile tags; `scripts/verify_agent_contract.py` | `docs/consumer-agent-guide.md` | | Diagnostic reporting and controlled reruns | `src/main/java/tools/dscode/common/reporting/diagnostic`; `InvestigationHandoff`; diagnostic aspects; `DiagnosticCli.java`; `emit-investigation`; `VisualFingerprintComparator.java`; `DiagnosticRunComparator.java` | `DiagnosticReportingChecks.java`; `Diagnostic213CompletionChecks.java`; `InvestigationHandoffChecks.java`; diagnostic features | `docs/diagnostic-reporting.md`; `docs/ai-diagnostic-reporting-plan.md`; `docs/ai-run-configuration.md`; root `AGENTS.md` | | Nested steps/block conditionals | search `Nested`, `Conditional`, `Block`, `Condition` in core implementation | `nested-and-block-conditionals.feature` | `docs/nested-steps.md`; `docs/block-conditionals.md` | | Component scenarios/reusable RUN/selectors/markers | `ModularScenarios.java`; `ScenarioStep.java`; `ScenarioStepData.java`; `StepBase.java`; `StepExtension.java`; `CurrentScenarioState.java`; `CucumberScanUtil.java`; search `finalizerSteps`, `RunSelection` | `component-scenarios.feature`; `reusable-scenario-selection.feature`; `run-step-parameter-variations.feature`; marker features | `docs/component-scenarios.md`; `docs/service-call-scenarios.md`; `docs/data-values-and-elements.md` | diff --git a/src/main/resources/META-INF/pickleball/guidance/docs/agent/repository-index.md b/src/main/resources/META-INF/pickleball/guidance/docs/agent/repository-index.md index 77f2a7b8..400252cc 100644 --- a/src/main/resources/META-INF/pickleball/guidance/docs/agent/repository-index.md +++ b/src/main/resources/META-INF/pickleball/guidance/docs/agent/repository-index.md @@ -604,6 +604,7 @@ This inventory helps coding agents discover relevant files. It does not replace ## Maven consumer feature scenarios +- `maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature` - `maven-consumer-project/src/test/resources/features/browser-action-contracts.feature` - `maven-consumer-project/src/test/resources/features/catalog-context.feature` - `maven-consumer-project/src/test/resources/features/component-scenarios.feature` diff --git a/src/main/resources/META-INF/pickleball/guidance/docs/consumer-agent-guide.md b/src/main/resources/META-INF/pickleball/guidance/docs/consumer-agent-guide.md index 18b9d23c..ea29adbc 100644 --- a/src/main/resources/META-INF/pickleball/guidance/docs/consumer-agent-guide.md +++ b/src/main/resources/META-INF/pickleball/guidance/docs/consumer-agent-guide.md @@ -295,3 +295,15 @@ Use `docs/consumer-project.md` on demand for the Maven consumer layout, local te If the consumer is nested inside the Pickleball source repository, repository-level `AGENTS.md` may impose additional maintainer rules for framework changes. Those core-maintainer rules are additive and do not replace this consumer-facing contract. For a normal external consumer, do not assume those core files exist. + +## Maintainer pointer-eval harness + +Pickleball's example Maven consumer includes an opt-in mixed pass/fail suite tagged only `@agent-pointer-eval` (`maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`). It is not part of `@all`, `@regression`, or the other Maven suite-profile tags. The failures are intentional canned fixtures for scoring whether a consumer AI agent follows the short `AGENTS.md` pointer into this guide and then uses discover-then-isolate. Do not treat those failures as product bugs, and do not "fix" the feature unless a human asked to change the harness. + +Run it explicitly: + +```text +mvn test -Dpkb_runvars="pkb_tags=@agent-pointer-eval, pkb_browser=CHROME_HEADLESS" +``` + +or `-Dpkb_tags=@agent-pointer-eval`. Do not add this tag to consumer `AGENTS.md` or Copilot pointer files. diff --git a/src/main/resources/META-INF/pickleball/guidance/index.txt b/src/main/resources/META-INF/pickleball/guidance/index.txt index 574e273e..1f0a8959 100644 --- a/src/main/resources/META-INF/pickleball/guidance/index.txt +++ b/src/main/resources/META-INF/pickleball/guidance/index.txt @@ -55,6 +55,7 @@ maven-consumer-project/src/test/resources/data/data-element-native-fixtures.feat maven-consumer-project/src/test/resources/data/data-reference-records.feature maven-consumer-project/src/test/resources/data/escaped-data-records.feature maven-consumer-project/src/test/resources/data/files/customerPayload.json +maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature maven-consumer-project/src/test/resources/features/browser-action-contracts.feature maven-consumer-project/src/test/resources/features/catalog-context.feature maven-consumer-project/src/test/resources/features/component-scenarios.feature