Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a repeatable MCP conformance baseline for WebCodex.
The CI job runs the upstream
modelcontextprotocol/conformancesuite at pinned commit7169291ec0b68eb370fddcd9947313ab0d5e4156for the2026-07-28and2025-11-25server profiles.This PR adds:
/mcpfixture using the normal WebCodex MCP HTTP routeNo production MCP behavior changes in this PR.
Baseline behavior
The gate keeps scored requirements separate from the referee's
not_scoredscenarios.For scored scenarios, known non-passing checks are matched by exact scenario/check ID, expected status, and a stable evidence fingerprint. A new failure, a changed failure reason or status, a missing required scenario, or a stale baseline entry fails CI.
Infrastructure failures are never accepted as protocol results. Timeouts, transport failures, abnormal runner exits, malformed or empty scenario output, and pinned-runner mismatches all fail the gate.
not_scoredscenarios are still required to run and are kept in the raw evidence, but their protocol verdicts remain informational.Optional-capability classifications also track the server's advertised capability snapshot, so advertising a previously absent capability forces the baseline to be reviewed.
Current baseline
The current non-passing checks mainly fall into three groups:
The currently recorded protocol findings include:
2026-07-28request metadata error-code handling2025-11-25tools/listoutput schema mismatchdata.uriwarningTwo focused tests also record existing 2026 result-shape gaps for later fixes: streamed artifact resource reads do not carry the normal cache hints, and completed Connector Tasks embed a tool result without the nested
resultType.Those fixes are intentionally left out of this PR.
CI
The
mcp-conformancejob:The current branch passes the full CI matrix, including the MCP conformance lane.
Scope
This PR is limited to test infrastructure, CI wiring, baseline data, focused tests, and documentation. It does not add conformance-only tools to the production registry or change MCP runtime behavior.