Skip to content

tier-audit docs coverage: evaluate the feature list against the spec revision an SDK targets #441

Description

@paulbalandan

The tier audit's documentation check scores SDKs against the canonical 48-feature list in .claude/skills/mcp-sdk-tier-audit/references/feature-list.md. The list carries no lifecycle metadata, and the rubric in docs-coverage-prompt.md allows only PASS, PARTIAL, or FAIL, so a feature the spec itself has removed still scores FAIL when an SDK neither implements nor documents it.

As of the 2026-07-28 revision, 11 of the 48 rows are removed or deprecated: 14 and 15 (resources/subscribe/unsubscribe, removed by SEP-2575), 23 (sampling, deprecated by SEP-2577), 29 (notifications/elicitation/complete, absent from the final 2026-07-28 schema), 30 and 31 (roots, deprecated by SEP-2577), 32 and 33 (logging, SEP-2577/SEP-2575), 36 (ping, removed by SEP-2575), and 39 and 40 (legacy HTTP+SSE, deprecated in 2025-03-26). An SDK that tracks the current revision is therefore capped at 37/48 and can never meet Tier 1's "all non-experimental features documented with examples", even when every feature it ships is documented with examples. An SDK that keeps deprecated surface scores higher, which inverts the incentive the SEP-2596 lifecycle is meant to create.

The referee already handles this correctly for conformance: scenario filtering by removedIn at --spec-version is what keeps a 2026-07-28 SDK from being failed on initialize or logging/setLevel. The docs rubric is the one check still scoring against the union of all revisions.

Proposal:

  1. Add lifecycle columns to feature-list.md (introduced/deprecated/removed, with the SEP that did it), mirroring the scenarios' removedIn metadata.
  2. Add a rule to docs-coverage-prompt.md: when the audited SDK targets spec revision R, a feature removed at or before R scores N/A and leaves the denominator. A feature deprecated at or before R that the SDK deliberately declines also scores N/A when the omission is documented with a stated reason.

Observed in practice auditing an SDK that targets 2026-07-28 only: 37/48 with zero PARTIAL rows and all 11 FAILs being removed or deprecated features, each already accounted for in the SDK's own feature index with the SEP that retired it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions