Skip to content

test(mcp): add 2026-07-28 conformance baseline - #1

Closed
ITSGJY wants to merge 30 commits into
mainfrom
test/mcp-2026-conformance
Closed

ITSGJY wants to merge 30 commits into
mainfrom
test/mcp-2026-conformance

Conversation

@ITSGJY

@ITSGJY ITSGJY commented Sep 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

Add a repeatable MCP conformance baseline for WebCodex.

The CI job runs the upstream modelcontextprotocol/conformance suite at pinned commit 7169291ec0b68eb370fddcd9947313ab0d5e4156 for the 2026-07-28 and 2025-11-25 server profiles.

This PR adds:

  • a test-only loopback /mcp fixture using the normal WebCodex MCP HTTP route
  • a pinned conformance runner built from the pinned Git object on every run
  • a report gate for reviewed non-passing checks
  • raw conformance reports as GitHub Actions artifacts
  • regression tests for the gate itself
  • documentation for updating the baseline

No production MCP behavior changes in this PR.

Baseline behavior

The gate keeps scored requirements separate from the referee's not_scored scenarios.

For scored scenarios, known non-passing checks are matched by exact scenario/check ID, expected status, and a stable evidence fingerprint. A new failure, a changed failure reason or status, a missing required scenario, or a stale baseline entry fails CI.

Infrastructure failures are never accepted as protocol results. Timeouts, transport failures, abnormal runner exits, malformed or empty scenario output, and pinned-runner mismatches all fail the gate. not_scored scenarios are still required to run and are kept in the raw evidence, but their protocol verdicts remain informational.

Optional-capability classifications also track the server's advertised capability snapshot, so advertising a previously absent capability forces the baseline to be reviewed.

Current baseline

The current non-passing checks mainly fall into three groups:

  • upstream scenarios that depend on named reference tools or resources not exposed by the WebCodex product registry
  • optional MCP capabilities that WebCodex does not advertise
  • a small set of protocol issues to fix separately

The currently recorded protocol findings include:

  • DNS rebinding protection
  • 2026-07-28 request metadata error-code handling
  • a 2025-11-25 tools/list output schema mismatch
  • the advisory SEP-2164 data.uri warning

Two focused tests also record existing 2026 result-shape gaps for later fixes: streamed artifact resource reads do not carry the normal cache hints, and completed Connector Tasks embed a tool result without the nested resultType.

Those fixes are intentionally left out of this PR.

CI

The mcp-conformance job:

  1. runs the report-gate regression tests
  2. builds the pinned referee from a fresh archived copy of the pinned Git commit
  3. builds and starts the isolated WebCodex fixture
  4. runs both dated profiles
  5. checks the raw reports against the reviewed baseline
  6. uploads the reports and logs

The current branch passes the full CI matrix, including the MCP conformance lane.

Scope

This PR is limited to test infrastructure, CI wiring, baseline data, focused tests, and documentation. It does not add conformance-only tools to the production registry or change MCP runtime behavior.

root and others added 28 commits September 2, 2026 08:57
@ITSGJY ITSGJY changed the title test(mcp): establish 2026-07-28 conformance baseline test(mcp): add 2026-07-28 conformance baseline Sep 16, 2026
@ITSGJY ITSGJY closed this Sep 20, 2026
@ITSGJY
ITSGJY deleted the test/mcp-2026-conformance branch September 20, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants