Skip to content

refactor(mcp): integrate evaluator with production service - #7

Open
Lauritz-Timm wants to merge 20 commits into
developfrom
feat/mcp-phase-1-evaluation
Open

refactor(mcp): integrate evaluator with production service#7
Lauritz-Timm wants to merge 20 commits into
developfrom
feat/mcp-phase-1-evaluation

Conversation

@Lauritz-Timm

@Lauritz-Timm Lauritz-Timm commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Refactors the MCP evaluator from the original standalone cleanroom/eval package into the workspace crate crates/icm-mcp-eval, while preserving the frozen evaluation contract.

What changed

  • Ordinary modern MCP scenarios use production icm_mcp::McpService, ConnectionState, JsonRpcMessage, and icm-store directly in-process.
  • Legacy byte-exact wire goldens, malformed/oversized transport frames, isolation, CLI/provider, proxy, HTTP, and topology scenarios remain isolated candidate-process tests.
  • Production protocol constants, metadata validation, lifecycle error codes, and capped framing are shared rather than duplicated in the evaluator.
  • Frozen contracts, fixtures, goldens, normalization rules, checksums, and the complete 294-scenario inventory remain unchanged.
  • Added workspace CI/design checks and updated evaluator/MCP architecture documentation.

This supersedes the original PR description that called the evaluator standalone and product-independent: fixture construction remains evaluator-owned and independent, while the evaluator package intentionally links only production icm-mcp and icm-store for its direct service lane.

Validation

  • cargo fmt --all -- --check
  • cargo clippy --locked --offline --workspace --all-targets -- -D warnings
  • cargo test --locked --offline --workspace --no-fail-fast (serial test threads for the existing marginal store performance assertion)
  • verify-design: 294 scenarios, 31 tools, 11 structured tools
  • Two-root baseline self-test: equal reports, 199 PASS / 95 UNSUPPORTED_BASELINE

The current candidate branch does not yet expose icm-cli provider; provider scenarios therefore remain explicitly unsupported in baseline mode and require a provider-capable candidate for full acceptance.

@Lauritz-Timm
Lauritz-Timm force-pushed the feat/mcp-phase-1-evaluation branch from 167f5fe to 226ba3d Compare August 17, 2026 07:23
@Lauritz-Timm Lauritz-Timm changed the title feat(mcp): add isolated compatibility evaluation suite refactor(mcp): integrate evaluator with production service Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant