test(mcp): add 2026-07-28 conformance baseline

.github/workflows/ci.yml

@@ -198,6 +198,37 @@ jobs:
             explicit_resume_advanced_compatibility_schema_and_metadata_are_retained \
             http_project_connector_lists_and_dispatches_only_canonical_capabilities
 
+  mcp-conformance:
+    # Dated MCP wire evidence is independent from ordinary unit/integration tests.
+    # The script pins the external referee and rejects missing or silently skipped
+    # coverage; known failures require exact check-level classification.
+    needs: contract
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+        with:
+          shared-key: linux-ci
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+      - name: Check conformance report gate
+        run: python3 scripts/tests/test_mcp_conformance_report.py
+      - name: Run pinned MCP dated-revision baseline
+        run: bash scripts/mcp_conformance.sh
+      - name: Upload MCP conformance evidence
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: mcp-conformance-${{ github.sha }}
+          path: target/mcp-conformance/reports
+          if-no-files-found: ignore
+          retention-days: 7
+
   test-linux-rust:
     # Complete Linux Rust coverage is mandatory before merge, including owner PRs.
     # Start it immediately in parallel with contract/frontend work; the stable
@@ -760,7 +791,7 @@ jobs:
           require_lane desktop "$NEEDS_DESKTOP" "$DESKTOP_RESULT"
 
   test-native:
-    needs: [contract, changes, test-docker-server, test-macos, test-windows]
+    needs: [contract, changes, mcp-conformance, test-docker-server, test-macos, test-windows]
     # Stable branch-protection context: daily native aggregates and the path-aware
     # Server image smoke must resolve according to classifier policy.
     if: always()
@@ -772,13 +803,14 @@ jobs:
           CONTRACT_RESULT: ${{ needs.contract.result }}
           CHANGES_RESULT: ${{ needs.changes.result }}
           NEEDS_DOCKER: ${{ needs.changes.outputs.needs_docker }}
+          MCP_CONFORMANCE_RESULT: ${{ needs.mcp-conformance.result }}
           DOCKER_RESULT: ${{ needs.test-docker-server.result }}
           MACOS_RESULT: ${{ needs.test-macos.result }}
           WINDOWS_RESULT: ${{ needs.test-windows.result }}
         run: |
           set -euo pipefail
-          echo "changes=$CHANGES_RESULT contract=$CONTRACT_RESULT docker=$DOCKER_RESULT macos=$MACOS_RESULT windows=$WINDOWS_RESULT needs_docker=$NEEDS_DOCKER"
-          if [ "$CHANGES_RESULT" != success ] || [ "$CONTRACT_RESULT" != success ]; then
+          echo "changes=$CHANGES_RESULT contract=$CONTRACT_RESULT mcp_conformance=$MCP_CONFORMANCE_RESULT docker=$DOCKER_RESULT macos=$MACOS_RESULT windows=$WINDOWS_RESULT needs_docker=$NEEDS_DOCKER"
+          if [ "$CHANGES_RESULT" != success ] || [ "$CONTRACT_RESULT" != success ] || [ "$MCP_CONFORMANCE_RESULT" != success ]; then
             echo "CI policy or contract classification did not succeed" >&2
             exit 1
           fi

docs/MCP_CONFORMANCE.md

@@ -0,0 +1,178 @@
+# MCP conformance baseline
+
+WebCodex tracks MCP wire compatibility with the upstream
+[`modelcontextprotocol/conformance`](https://github.com/modelcontextprotocol/conformance)
+referee in addition to its focused Rust tests. The baseline is evidence about the
+checks that actually ran; it is not a blanket "MCP compliant" badge.
+
+## Pinned referee and profiles
+
+`scripts/mcp_conformance.sh` pins the upstream referee to the immutable commit in
+`tests/fixtures/mcp/conformance/baseline.json`. The script rejects an external
+checkout at a different commit, archives the pinned Git object into a fresh
+script-owned build directory, and runs `npm ci` plus the referee build from that
+archive every time. Existing `dist/` output and working-tree modifications are
+therefore never trusted as pinned evidence. Updating the pin is a reviewed
+test-contract change, not a floating dependency update.
+
+The ordinary baseline runs two dated server profiles independently:
+
+- `2026-07-28`, using the stateless per-request protocol and that revision's
+  frozen requirements;
+- `2025-11-25`, using the stateful lifecycle and its reconstructed frozen
+  requirements.
+
+Focused repository tests continue to cover `2025-06-18`; the conformance harness
+is not used to invent a frozen requirements set that it does not publish.
+
+Run the same baseline locally with:
+
+```bash
+bash scripts/mcp_conformance.sh
+```
+
+To reuse an already checked-out referee without changing the pin:
+
+```bash
+WEBCODEX_MCP_CONFORMANCE_HARNESS_DIR=/path/to/conformance \
+  bash scripts/mcp_conformance.sh 2026-07-28
+```
+
+The supplied checkout must resolve to the exact pinned commit. A checkout supplied
+through `WEBCODEX_MCP_CONFORMANCE_HARNESS_DIR` is read-only from this script's
+perspective: invalid or mismatched checkouts are rejected rather than deleted or
+rewritten, and local working-tree changes are ignored by the pinned `git archive`.
+Reports are written to `target/mcp-conformance/reports` by default. An explicitly
+supplied report directory must be empty or carry the WebCodex ownership marker
+before old profile output is removed. CI uploads the report directory even when
+the gate fails so the raw evidence remains inspectable.
+
+## What is exercised
+
+The script starts one ignored, test-only Rust fixture on `127.0.0.1:0`. The
+fixture composes the same Salvo `AuthMiddleware` and `/mcp` handlers used by the
+production HTTP surface, with temporary database/runtime state and authentication
+disabled in the isolated test config. The referee therefore reaches real WebCodex parsing, dispatch and
+rendering without production credentials or an external deployment.
+
+No conformance-only tool is added to the production registry. Some upstream
+scenarios intentionally expect named reference-fixture tools. When WebCodex does
+not expose such a tool, that result is a coverage gap until a test-only adapter is
+provided; it must not be relabeled as a product protocol failure merely to make a
+summary green.
+
+Authenticated Runtime/Connector behavior remains covered by focused synthetic-
+credential integration tests. The upstream server CLI does not provide a generic
+way to inject WebCodex authorization headers, so an authentication-blocked
+scenario is inconclusive rather than a pass.
+
+## Coverage-aware report gate
+
+The upstream runner saves one `checks.json` file per scenario. After each profile,
+`scripts/mcp_conformance_report.py` verifies the raw reports against the exact
+required server-scenario list and the reviewed baseline.
+
+Every `FAILURE`, `WARNING`, or `SKIPPED` check from a **scored required**
+scenario must have an exact `<scenario>, <check_id>` classification in
+`tests/fixtures/mcp/conformance/baseline.json`. Each classification also pins the
+observed status and a SHA-256 fingerprint of the stable failure evidence
+(`errorMessage` plus structured details/metadata when present, falling back to
+the check's name/description). If the same check ID starts failing for a different reason or
+with a different severity, the gate requires review instead of reusing the old
+classification. Whole-scenario wildcards are rejected.
+
+Classifications that rely on an optional capability also bind to the fixture's
+actual capability advertisement. Before the referee run, the script probes the
+same HTTP endpoint with `server/discover` for `2026-07-28` and `initialize` for
+`2025-11-25`, keeps the relevant tools/resources/prompts/logging/completions
+capabilities, and compares them with the profile snapshot in the baseline. If the
+server starts advertising a previously absent capability, those optional
+classifications require review even when the old check ID still fails.
+
+The referee's `not_scored` server scenarios are still required to run and are
+retained as informational protocol evidence. Their protocol verdicts do not
+consume expected-failure classifications, but infrastructure failures such as a
+scenario timeout or connection failure remain gate-blocking. Supported scored
+classifications are:
+
+| Classification | Meaning |
+|---|---|
+| `genuine_protocol_failure` | The intended WebCodex behavior was reached and a protocol requirement is known to be violated. |
+| `missing_harness_fixture` | The upstream scenario depends on reference-fixture behavior that the product endpoint does not expose. The product behavior remains untested by that check. |
+| `optional_capability_not_implemented` | A capability is genuinely optional and is not advertised/implemented. The entry must cite capability/spec evidence. |
+| `harness_limitation_pending` | The pinned referee itself marks or demonstrates a limitation/pending check. |
+| `inconclusive_infrastructure` | Authentication blocking, timeout, harness exception, or other infrastructure failure. This is recorded but **never satisfies the CI gate**. |
+
+The gate also fails when:
+
+- no applicable `SUCCESS`/`FAILURE`/`WARNING` checks were emitted;
+- an individual required scenario emits no verdict (for example an empty or
+  INFO-only report);
+- a scored or `not_scored` scenario report expected by the frozen requirements is
+  missing, duplicated, or an unknown scenario is unexpectedly present;
+- a check contains an unknown status;
+- a new non-success check in a scored required scenario has no exact classification;
+- a classified check changes status/evidence, disappears, or now passes;
+- the referee exits abnormally, its exit code disagrees with the scored FAILURE
+  set, or any scored/not-scored scenario emits `scenario-timeout`,
+  `wire-schema-harness-error`, one of the pinned referee's generic top-level
+  scenario-error checks, the outer `Failed to run scenario` shape, or an exact
+  transport-exception form emitted by the pinned Node runner;
+- the server capability advertisement changes while optional-capability baseline
+  entries depend on the previous advertisement;
+- an explicitly classified inconclusive infrastructure result is present.
+
+For the pinned referee, exit `0` means no scored FAILURE and exit `1` means at
+least one scored FAILURE; other exit codes are treated as runner/infrastructure
+failures. This is stricter than trusting the process exit code alone because the
+raw reports still need complete coverage and reviewed classifications.
+
+The report gate's own regression suite is dependency-free:
+
+```bash
+python3 scripts/tests/test_mcp_conformance_report.py
+```
+
+It covers all-skipped output, missing/empty/INFO-only scenarios, unknown statuses,
+abnormal referee exits, scored and not-scored infrastructure/transport failures,
+changed status/evidence for an existing check ID, capability-advertisement drift,
+ambiguous duplicate non-success check IDs, harness-pin mismatch, placeholder evidence, new scored
+failures, informational non-success reporting, stale classifications, exact-path
+JSON-RPC request-ID normalization, wire-schema payload drift, ordinary lookalike
+objects, empty structured evidence, and rejection of broad masks.
+
+## Raw evidence and review policy
+
+Each profile retains:
+
+- the referee's raw per-scenario `checks.json` files;
+- the referee stdout/stderr log;
+- metadata containing the WebCodex source SHA, referee SHA, profile, upstream
+  referee exit code, observed server capability projection, exact scored scenario
+  list, and frozen `not_scored` server scenarios with their upstream reasons;
+- the coverage-aware WebCodex summary.
+
+Baseline updates should be narrow. Add a classification only after reproducing
+and understanding the exact check, then record its observed status and evidence
+fingerprint. Fingerprints retain semantic diagnostics but normalize two narrowly
+scoped referee noise sources: the numeric value (not presence/type) of the two
+known SEP-2575 millisecond-epoch response IDs, and the duplicated offending
+message only inside the structured `wire-schema-valid` violation list. For a
+`ListToolsResult/tools/<index>/...` diagnostic, that volatile array ordinal is
+resolved through the offending response to the referenced tool name before hashing,
+so unrelated tool insertion/reordering does not rename the finding; moving the defect
+to a different tool still changes the fingerprint. Wire-schema origin/context/errors/
+specVersion remain hashed, and ordinary objects with similar field names are not
+normalized. A changed fingerprint is a review prompt, not something
+to refresh mechanically. Remove the entry as soon as the check passes. Do not replace
+multiple check IDs with a scenario-wide exception. The SEP-2164 `data.uri`
+finding is kept as an advisory SHOULD/WARNING rather than described as a MUST
+violation.
+
+This baseline deliberately records two already-demonstrated 2026 result-shape findings
+without fixing production behavior: streamed artifact `resources/read` omits the
+cache hints emitted by ordinary resource reads, and a completed Connector Task
+embeds its original tool result without a nested `resultType`. Their focused Rust
+observations and planned follow-up are recorded in the baseline fixture. Fixes
+belong in a separate protocol-behavior change so this baseline remains an
+independent measuring instrument.

docs/TESTING.md

@@ -33,6 +33,7 @@ waits, and the cost of each test lane.
 | real-process job reconciliation harness | Boot a real server plus a WebSocket runner that advertises `job_state_reconciliation`. Scenario A keeps a raw async Job running across a SERVER-only restart and asserts the SAME runner instance, original `job_id`, preserved ownership/project/session, non-regressing sequence/log cursors, `recovered_after_server_restart`, original-process stop, and one side-effect set. Scenario B lets a Job complete while the Server is offline and reconciles the terminal result without duplicate logs or execution. Scenario C forces `run_process` past its synchronous grace window, then proves the handed-off structured Job survives a Server restart and an old Server-epoch observation token refreshes immediately for the same `job_id`. Scenario D uses a delayed Cargo fixture to force a real `cargo_check` validation handoff past its sync window, then proves the same restart/token-refresh/stop contract with the validation command started exactly once. Ordinary Runner-owned Jobs keep the Runner process alive for these scenarios; `run_detached_process` restart survival is a separate supervisor-ownership contract covered by its focused Runner suites and production dogfood. | Local processes, temp dirs/ports/tokens, and a temp project; no production services or QUIC certs. Scenario D intentionally takes roughly the validation sync window plus restart time. | `bash scripts/e2e_job_reconciliation_ws.sh` |
 | real-process job recovery failure/non-reconciliation harness | Cover the failure and non-reconciliation paths the happy-path reconciliation harness omits, using `WEBCODEX_JOB_RECOVERY_GRACE_SECS=10` (clamped, above the 5s floor) so the deadline is bounded without waiting the 120s default. Scenario C: kill the runner only (server stays up), let the job enter `recovering`, and assert the non-request-triggered recovery-timeout sweep transitions it to `lost` with `runner_recovery_deadline_exceeded`, `ended_at` set once, one list record, stop-on-lost stable, and the command never re-executes. Scenario D: instance B replaces instance A (same client_id, new `agent_instance_id`); A's job becomes `lost` with `runner_instance_replaced`, B starts its own new job, A's late update is rejected, first `ended_at`/reason preserved. Scenario E: a generation-2 Runner registered with `WEBCODEX_RUNNER_DISABLE_JOB_STATE_RECONCILIATION=1` (no capability, no inventory) dispatches a job and, on disconnect, deterministically fences it to `lost` with `runner_disconnected_without_reconciliation` (never `recovering`); after a server restart its public lost receipt remains observable within retention, and a same-client new no-reconciliation instance cannot revive execution. Scenario F: a long job across three server restarts keeps the same `job_id`, runs the command once, keeps `last_update_seq`/log cursors non-regressing and markers non-duplicating, and reaches a terminal `stopped` that survives a third restart with `ended_at` unchanged by terminal inventory replay. | Local processes, temp dirs/ports/tokens, and a temp project; no production services or QUIC certs. | `bash scripts/e2e_job_recovery_failures_ws.sh` |
 | security auth matrix | Cover OAuth, scope policy, shared-key behavior, token classes, read-only session guards, and denied mutations. | No external identity provider by default; use local fixtures and synthetic tokens. | `cargo test -p webcodex --lib oauth -- --nocapture`; `cargo test -p webcodex --lib scope -- --nocapture`; `cargo test -p webcodex --lib metadata -- --nocapture` |
+| MCP conformance baseline | Exercise the real Salvo `/mcp` handlers through a disposable loopback fixture against a pinned upstream referee, then gate on check-level coverage/classification rather than process exit alone. | Loopback product fixture; immutable external referee source/dependencies fetched only by the explicit script/CI lane; no production credentials. | `bash scripts/mcp_conformance.sh`; see [`MCP_CONFORMANCE.md`](MCP_CONFORMANCE.md). |
 
 The MCP Apps' DOM/message-order regression tests run without browser or npm
 dependencies: `node --test src/mcp_tests/*.test.mjs`. They execute the embedded
@@ -95,6 +96,14 @@ The lanes above define test semantics; workflows decide when to run them.
   `test-windows`, and `test-native` aggregates always resolve and verify each child
   lane is `success` when required or `skipped` when not required, avoiding a skipped
   required-check context that could leave branch protection pending.
+- MCP dated-revision evidence has its own bounded `mcp-conformance` lane. It pins
+  and freshly builds the upstream referee, runs the `2026-07-28` and `2025-11-25`
+  server requirements against a test-only loopback WebCodex endpoint, validates
+  the report gate itself, and uploads raw reports on success or failure.
+  `test-native` requires this lane to succeed, so missing per-scenario verdicts,
+  abnormal/infrastructure runs, stale or changed classifications, and unclassified
+  new failures are merge-blocking. See [`MCP_CONFORMANCE.md`](MCP_CONFORMANCE.md)
+  for baseline semantics.
 - Linux Rust execution remains package-sharded: the server package `webcodex`, the
   Runner/LSP packages, and the remaining workspace crates run in parallel. The
   Runner/LSP shard compiles with `--features runner-real-process-tests` to prevent