Add opt-in @agent-pointer-eval mixed harness for consumer AI pointer tests

docs/agent/feature-map.md

@@ -20,6 +20,7 @@ This file maps consumer-visible capabilities to implementation anchors, executab
 | Mapping, ParsingMap/NodeMap, templates/directives | `MappingSteps.java`; `FileAndDataParsing.java`; `MappingProcessor.java`; `QuoteParser.java`; `NodeMap.java`; `ParsingMap.java`; `ValueFormatting.java`; `common/dataelements` | `mapping-and-resources.feature`; `mapping-value-type-preservation.feature`; `scenario-data-references.feature`; Data Element features; `QuoteParserChecks.java`; internal Java checks | `docs/mapping-and-templating.md`; `docs/data-values-and-elements.md`; `docs/data-element-query-runtime.md`; `docs/config-files-and-resource-mapping.md` |
 | Configuration/profiles/RunVars | `PKB_props.java`; `PickleballProfiles.java`; `PkbPropertyValueNormalizer.java`; runner/config classes; search `pkb_profile`, `pkb_runvars`, `pkb_run_profile`, `pkb_configpath` | `configuration-system-properties.feature`; `ProfileConfigurationChecks.java`; consumer properties/profile examples | `docs/configuration.md`; `docs/getting-started.md`; `docs/ai-run-configuration.md`; `docs/consumer-project.md` |
 | Consumer guidance export/reference snapshot | `DiagnosticCli.java`; `gradle/consumer-guidance.gradle`; `scripts/sync_consumer_guidance.py`; `maven-consumer-project/AGENTS.md`; `maven-consumer-project/.github/copilot-instructions.md`; search `export-guidance`, `discover-hint`, `GUIDANCE-MANIFEST.json`, `.pickleball/investigations` | `PickleballGuidanceChecks.java`; consumer guidance contract checks | `docs/consumer-agent-guide.md`; `docs/consumer-project.md` |
+| Consumer AI-agent pointer-eval harness | `maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`; tag `@agent-pointer-eval` (optional `@agent-pointer-eval-pass` / `@agent-pointer-eval-fail`) | opt-in mixed pass/fail; must not carry `@all`, `@regression`, or other Maven suite-profile tags; `scripts/verify_agent_contract.py` | `docs/consumer-agent-guide.md` |
 | Diagnostic reporting and controlled reruns | `src/main/java/tools/dscode/common/reporting/diagnostic`; `InvestigationHandoff`; diagnostic aspects; `DiagnosticCli.java`; `emit-investigation`; `VisualFingerprintComparator.java`; `DiagnosticRunComparator.java` | `DiagnosticReportingChecks.java`; `Diagnostic213CompletionChecks.java`; `InvestigationHandoffChecks.java`; diagnostic features | `docs/diagnostic-reporting.md`; `docs/ai-diagnostic-reporting-plan.md`; `docs/ai-run-configuration.md`; root `AGENTS.md` |
 | Nested steps/block conditionals | search `Nested`, `Conditional`, `Block`, `Condition` in core implementation | `nested-and-block-conditionals.feature` | `docs/nested-steps.md`; `docs/block-conditionals.md` |
 | Component scenarios/reusable RUN/selectors/markers | `ModularScenarios.java`; `ScenarioStep.java`; `ScenarioStepData.java`; `StepBase.java`; `StepExtension.java`; `CurrentScenarioState.java`; `CucumberScanUtil.java`; search `finalizerSteps`, `RunSelection` | `component-scenarios.feature`; `reusable-scenario-selection.feature`; `run-step-parameter-variations.feature`; marker features | `docs/component-scenarios.md`; `docs/service-call-scenarios.md`; `docs/data-values-and-elements.md` |

docs/agent/repository-index.md

@@ -604,6 +604,7 @@ This inventory helps coding agents discover relevant files. It does not replace
 
 ## Maven consumer feature scenarios
 
+- `maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`
 - `maven-consumer-project/src/test/resources/features/browser-action-contracts.feature`
 - `maven-consumer-project/src/test/resources/features/catalog-context.feature`
 - `maven-consumer-project/src/test/resources/features/component-scenarios.feature`

docs/consumer-agent-guide.md

@@ -295,3 +295,15 @@ Use `docs/consumer-project.md` on demand for the Maven consumer layout, local te
 If the consumer is nested inside the Pickleball source repository, repository-level `AGENTS.md` may impose additional maintainer rules for framework changes. Those core-maintainer rules are additive and do not replace this consumer-facing contract.
 
 For a normal external consumer, do not assume those core files exist.
+
+## Maintainer pointer-eval harness
+
+Pickleball's example Maven consumer includes an opt-in mixed pass/fail suite tagged only `@agent-pointer-eval` (`maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`). It is not part of `@all`, `@regression`, or the other Maven suite-profile tags. The failures are intentional canned fixtures for scoring whether a consumer AI agent follows the short `AGENTS.md` pointer into this guide and then uses discover-then-isolate. Do not treat those failures as product bugs, and do not "fix" the feature unless a human asked to change the harness.
+
+Run it explicitly:
+
+```text
+mvn test -Dpkb_runvars="pkb_tags=@agent-pointer-eval, pkb_browser=CHROME_HEADLESS"
+```
+
+or `-Dpkb_tags=@agent-pointer-eval`. Do not add this tag to consumer `AGENTS.md` or Copilot pointer files.

maven-consumer-project/src/test/java/tools/dscode/common/reporting/diagnostic/PickleballGuidanceChecks.java

@@ -79,6 +79,8 @@ void dependencyPrintsCanonicalAgentGuide() {
             assertTrue(chooserList.contains("run-catalog.json"));
             assertTrue(chooserList.contains("after discovery has named the trouble spots"));
             assertTrue(chooserList.contains("Do not start a worker just to run the whole suite"));
+            assertTrue(guide.contains("Maintainer pointer-eval harness"));
+            assertTrue(guide.contains("@agent-pointer-eval"));
             String liveLoop = guide.substring(
                     guide.indexOf("### Live isolation loop"),
                     guide.indexOf("Generated guidance lifecycle")
@@ -200,6 +202,9 @@ void dependencyExportsVersionMatchedGuidanceAndManifest() throws Exception {
             assertTrue(Files.isRegularFile(root.resolve(
                     "maven-consumer-project/src/test/resources/features/dynamic-steps.feature"
             )));
+            assertTrue(Files.isRegularFile(root.resolve(
+                    "maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature"
+            )));
             assertTrue(Files.isRegularFile(root.resolve(
                     "maven-consumer-project/src/test/resources/calls/service-call-definitions.feature"
             )));
@@ -237,6 +242,9 @@ void dependencyExportsVersionMatchedGuidanceAndManifest() throws Exception {
             assertTrue(managedFiles.contains(
                     "maven-consumer-project/src/test/resources/features/dynamic-steps.feature"
             ));
+            assertTrue(managedFiles.contains(
+                    "maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature"
+            ));
             assertTrue(managedFiles.stream().noneMatch(path -> path.contains("_local2")));
             assertFalse(managedFiles.contains("maven-consumer-project/AGENTS.md"));
             assertFalse(managedFiles.contains("maven-consumer-project/.github/copilot-instructions.md"));
@@ -260,6 +268,8 @@ void dependencyExportsVersionMatchedGuidanceAndManifest() throws Exception {
             assertTrue(guide.contains("tell the human"));
             assertTrue(guide.contains("CLI/Maven-primary"));
             assertTrue(guide.contains("PickleballWorkbenchLauncher"));
+            assertTrue(guide.contains("Maintainer pointer-eval harness"));
+            assertTrue(guide.contains("@agent-pointer-eval"));
 
             String consumerProject = Files.readString(root.resolve("docs/consumer-project.md"));
             assertTrue(consumerProject.contains("keep console verbosity low"));

maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature

@@ -0,0 +1,32 @@
+@agent-pointer-eval
+Feature: Agent pointer-eval mixed harness
+  Intentional mixed pass/fail harness for consumer AI-agent pointer tests.
+  It is not part of @all or @regression. Run it only with pkb_tags=@agent-pointer-eval
+  (or the pkb_runvars equivalent). Failures are canned and stable. Do not "fix"
+  this file as if it were product coverage unless the human asked to change the harness.
+
+  @agent-pointer-eval @agent-pointer-eval-pass
+  Scenario: Agent pointer eval passing pickleball equality
+    * , verify "pickleball" equals "pickleball"
+
+  @agent-pointer-eval @agent-pointer-eval-pass
+  Scenario: Agent pointer eval passing pointer token equality
+    * , verify "pointer-eval" equals "pointer-eval"
+
+  @agent-pointer-eval @agent-pointer-eval-fail
+  Scenario: Agent pointer eval failing fruit mismatch
+    * , verify "apple" equals "orange"
+
+  @agent-pointer-eval @agent-pointer-eval-fail
+  Scenario: Agent pointer eval failing greek-letter mismatch
+    * , verify "alpha" equals "omega"
+
+  @agent-pointer-eval @agent-pointer-eval-pass
+  Scenario: Agent pointer eval passing home heading
+    * navigate to: URL.home
+    * , ensure "Pickleball Test Lab" Text is displayed
+
+  @agent-pointer-eval @agent-pointer-eval-fail
+  Scenario: Agent pointer eval failing forms heading
+    * navigate to: URL.forms
+    * , ensure "AGENT POINTER EVAL INTENTIONAL FORMS HEADING" Text is displayed

scripts/verify_agent_contract.py

@@ -346,6 +346,7 @@ def validate_consumer_bridge(errors: list[str]) -> None:
             "Diagnostic investigation protocol",
             "attach.json",
             "ui .",
+            "@agent-pointer-eval",
         ):
             if forbidden in text:
                 errors.append(
@@ -400,6 +401,7 @@ def validate_dependency_owned_guidance(errors: list[str]) -> None:
         "version-matched",
         "Generated Maven consumer reference",
         "maven-consumer-project/",
+        "@agent-pointer-eval",
     ):
         if required not in text:
             errors.append(
@@ -449,6 +451,98 @@ def validate_consumer_ignore(errors: list[str]) -> None:
         )
 
 
+AGENT_POINTER_EVAL_FEATURE = (
+    "maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature"
+)
+AGENT_POINTER_EVAL_TAG = "@agent-pointer-eval"
+MAVEN_SUITE_PROFILE_TAGS = {
+    "@all",
+    "@regression",
+    "@smoke",
+    "@browser",
+    "@data",
+    "@navigation",
+    "@forms",
+    "@catalog",
+    "@mapping",
+    "@resources",
+    "@workflow",
+    "@block-conditionals",
+    "@nested-steps",
+    "@keyboard",
+    "@dialogs",
+    "@components",
+}
+
+
+def parse_gherkin_tagged_elements(text: str) -> list[tuple[str, list[str]]]:
+    pending: list[str] = []
+    elements: list[tuple[str, list[str]]] = []
+    for raw in text.splitlines():
+        stripped = raw.strip()
+        if not stripped or stripped.startswith("#"):
+            continue
+        if stripped.startswith("@"):
+            pending.extend(token for token in stripped.split() if token.startswith("@"))
+            continue
+        if (
+            stripped.startswith("Feature:")
+            or stripped.startswith("Scenario:")
+            or stripped.startswith("Scenario Outline:")
+        ):
+            elements.append((stripped, pending))
+            pending = []
+    return elements
+
+
+def validate_agent_pointer_eval_harness(errors: list[str]) -> None:
+    relative = AGENT_POINTER_EVAL_FEATURE
+    path = ROOT / relative
+    if not path.is_file():
+        errors.append("Missing opt-in consumer pointer-eval harness: " + relative)
+        return
+
+    elements = parse_gherkin_tagged_elements(path.read_text(encoding="utf-8"))
+    if not elements or not elements[0][0].startswith("Feature:"):
+        errors.append("Pointer-eval harness must declare a Feature: " + relative)
+        return
+
+    scenarios = [
+        element
+        for element in elements
+        if element[0].startswith("Scenario:") or element[0].startswith("Scenario Outline:")
+    ]
+    if len(scenarios) < 2:
+        errors.append(
+            "Pointer-eval harness must contain more than one Scenario so isolate can "
+            "target distinct names: " + relative
+        )
+
+    for name, tags in elements:
+        missing = AGENT_POINTER_EVAL_TAG not in tags
+        forbidden = sorted(tag for tag in tags if tag in MAVEN_SUITE_PROFILE_TAGS)
+        extras = [
+            tag
+            for tag in tags
+            if tag != AGENT_POINTER_EVAL_TAG and not tag.startswith(AGENT_POINTER_EVAL_TAG + "-")
+        ]
+        if missing:
+            errors.append(
+                f"Pointer-eval harness element must carry {AGENT_POINTER_EVAL_TAG}: "
+                f"{relative} -> {name}"
+            )
+        if forbidden:
+            errors.append(
+                "Pointer-eval harness must not carry Maven suite-profile tags "
+                f"{forbidden}: {relative} -> {name}"
+            )
+        if extras:
+            errors.append(
+                "Pointer-eval harness may only add unique @agent-pointer-eval-* tags, "
+                f"not {extras}: {relative} -> {name}"
+            )
+
+
 def validate_packaged_guidance(errors: list[str]) -> None:
     script = ROOT / "scripts" / "sync_consumer_guidance.py"
     if not script.is_file():
@@ -542,6 +636,7 @@ def main() -> int:
     validate_consumer_bridge(errors)
     validate_consumer_readme(errors)
     validate_dependency_owned_guidance(errors)
+    validate_agent_pointer_eval_harness(errors)
     validate_consumer_reference_build_hook(errors)
     validate_consumer_tracked_artifacts(errors)
     validate_consumer_ignore(errors)

src/main/resources/META-INF/pickleball/guidance/AGENT-GUIDE.md

@@ -295,3 +295,15 @@ Use `docs/consumer-project.md` on demand for the Maven consumer layout, local te
 If the consumer is nested inside the Pickleball source repository, repository-level `AGENTS.md` may impose additional maintainer rules for framework changes. Those core-maintainer rules are additive and do not replace this consumer-facing contract.
 
 For a normal external consumer, do not assume those core files exist.
+
+## Maintainer pointer-eval harness
+
+Pickleball's example Maven consumer includes an opt-in mixed pass/fail suite tagged only `@agent-pointer-eval` (`maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`). It is not part of `@all`, `@regression`, or the other Maven suite-profile tags. The failures are intentional canned fixtures for scoring whether a consumer AI agent follows the short `AGENTS.md` pointer into this guide and then uses discover-then-isolate. Do not treat those failures as product bugs, and do not "fix" the feature unless a human asked to change the harness.
+
+Run it explicitly:
+
+```text
+mvn test -Dpkb_runvars="pkb_tags=@agent-pointer-eval, pkb_browser=CHROME_HEADLESS"
+```
+
+or `-Dpkb_tags=@agent-pointer-eval`. Do not add this tag to consumer `AGENTS.md` or Copilot pointer files.

src/main/resources/META-INF/pickleball/guidance/docs/agent/feature-map.md

@@ -20,6 +20,7 @@ This file maps consumer-visible capabilities to implementation anchors, executab
 | Mapping, ParsingMap/NodeMap, templates/directives | `MappingSteps.java`; `FileAndDataParsing.java`; `MappingProcessor.java`; `QuoteParser.java`; `NodeMap.java`; `ParsingMap.java`; `ValueFormatting.java`; `common/dataelements` | `mapping-and-resources.feature`; `mapping-value-type-preservation.feature`; `scenario-data-references.feature`; Data Element features; `QuoteParserChecks.java`; internal Java checks | `docs/mapping-and-templating.md`; `docs/data-values-and-elements.md`; `docs/data-element-query-runtime.md`; `docs/config-files-and-resource-mapping.md` |
 | Configuration/profiles/RunVars | `PKB_props.java`; `PickleballProfiles.java`; `PkbPropertyValueNormalizer.java`; runner/config classes; search `pkb_profile`, `pkb_runvars`, `pkb_run_profile`, `pkb_configpath` | `configuration-system-properties.feature`; `ProfileConfigurationChecks.java`; consumer properties/profile examples | `docs/configuration.md`; `docs/getting-started.md`; `docs/ai-run-configuration.md`; `docs/consumer-project.md` |
 | Consumer guidance export/reference snapshot | `DiagnosticCli.java`; `gradle/consumer-guidance.gradle`; `scripts/sync_consumer_guidance.py`; `maven-consumer-project/AGENTS.md`; `maven-consumer-project/.github/copilot-instructions.md`; search `export-guidance`, `discover-hint`, `GUIDANCE-MANIFEST.json`, `.pickleball/investigations` | `PickleballGuidanceChecks.java`; consumer guidance contract checks | `docs/consumer-agent-guide.md`; `docs/consumer-project.md` |
+| Consumer AI-agent pointer-eval harness | `maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`; tag `@agent-pointer-eval` (optional `@agent-pointer-eval-pass` / `@agent-pointer-eval-fail`) | opt-in mixed pass/fail; must not carry `@all`, `@regression`, or other Maven suite-profile tags; `scripts/verify_agent_contract.py` | `docs/consumer-agent-guide.md` |
 | Diagnostic reporting and controlled reruns | `src/main/java/tools/dscode/common/reporting/diagnostic`; `InvestigationHandoff`; diagnostic aspects; `DiagnosticCli.java`; `emit-investigation`; `VisualFingerprintComparator.java`; `DiagnosticRunComparator.java` | `DiagnosticReportingChecks.java`; `Diagnostic213CompletionChecks.java`; `InvestigationHandoffChecks.java`; diagnostic features | `docs/diagnostic-reporting.md`; `docs/ai-diagnostic-reporting-plan.md`; `docs/ai-run-configuration.md`; root `AGENTS.md` |
 | Nested steps/block conditionals | search `Nested`, `Conditional`, `Block`, `Condition` in core implementation | `nested-and-block-conditionals.feature` | `docs/nested-steps.md`; `docs/block-conditionals.md` |
 | Component scenarios/reusable RUN/selectors/markers | `ModularScenarios.java`; `ScenarioStep.java`; `ScenarioStepData.java`; `StepBase.java`; `StepExtension.java`; `CurrentScenarioState.java`; `CucumberScanUtil.java`; search `finalizerSteps`, `RunSelection` | `component-scenarios.feature`; `reusable-scenario-selection.feature`; `run-step-parameter-variations.feature`; marker features | `docs/component-scenarios.md`; `docs/service-call-scenarios.md`; `docs/data-values-and-elements.md` |

src/main/resources/META-INF/pickleball/guidance/docs/agent/repository-index.md

@@ -604,6 +604,7 @@ This inventory helps coding agents discover relevant files. It does not replace
 
 ## Maven consumer feature scenarios
 
+- `maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`
 - `maven-consumer-project/src/test/resources/features/browser-action-contracts.feature`
 - `maven-consumer-project/src/test/resources/features/catalog-context.feature`
 - `maven-consumer-project/src/test/resources/features/component-scenarios.feature`

src/main/resources/META-INF/pickleball/guidance/docs/consumer-agent-guide.md

@@ -295,3 +295,15 @@ Use `docs/consumer-project.md` on demand for the Maven consumer layout, local te
 If the consumer is nested inside the Pickleball source repository, repository-level `AGENTS.md` may impose additional maintainer rules for framework changes. Those core-maintainer rules are additive and do not replace this consumer-facing contract.
 
 For a normal external consumer, do not assume those core files exist.
+
+## Maintainer pointer-eval harness
+
+Pickleball's example Maven consumer includes an opt-in mixed pass/fail suite tagged only `@agent-pointer-eval` (`maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature`). It is not part of `@all`, `@regression`, or the other Maven suite-profile tags. The failures are intentional canned fixtures for scoring whether a consumer AI agent follows the short `AGENTS.md` pointer into this guide and then uses discover-then-isolate. Do not treat those failures as product bugs, and do not "fix" the feature unless a human asked to change the harness.
+
+Run it explicitly:
+
+```text
+mvn test -Dpkb_runvars="pkb_tags=@agent-pointer-eval, pkb_browser=CHROME_HEADLESS"
+```
+
+or `-Dpkb_tags=@agent-pointer-eval`. Do not add this tag to consumer `AGENTS.md` or Copilot pointer files.

src/main/resources/META-INF/pickleball/guidance/index.txt

@@ -55,6 +55,7 @@ maven-consumer-project/src/test/resources/data/data-element-native-fixtures.feat
 maven-consumer-project/src/test/resources/data/data-reference-records.feature
 maven-consumer-project/src/test/resources/data/escaped-data-records.feature
 maven-consumer-project/src/test/resources/data/files/customerPayload.json
+maven-consumer-project/src/test/resources/features/agent-pointer-eval.feature
 maven-consumer-project/src/test/resources/features/browser-action-contracts.feature
 maven-consumer-project/src/test/resources/features/catalog-context.feature
 maven-consumer-project/src/test/resources/features/component-scenarios.feature