Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,8 @@ jobs:
- run: npm run build
- run: node --test --test-reporter=spec test/mcp.test.mjs

# Dogfood: the Action from this checkout, on its own fixture server.
# Dogfood: the Action from this checkout, on its own fixture server, then on toolmenu
# mcp (the PR comment: one per PR, so only the second run posts it).
action:
runs-on: ubuntu-latest
permissions:
Expand All @@ -69,6 +70,13 @@ jobs:
command: node test/fixtures/sdk-server.mjs
baseline: examples/menu.json
release: 'off' # package.json is toolmenu's version, not the fixture server's
comment: 'false'
- uses: ./
with:
cli: node dist/cli.js
command: node dist/cli.js mcp
baseline: docs/toolmenu-menu.json
release: 'off' # the version is bumped at release, not in each PR

# The Action on a job whose Node is older than 22: toolmenu gets its own Node 22,
# and the server and the job's later steps keep Node 20.
Expand Down
7 changes: 5 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,9 @@ findings on every run.
- `docs/snapshot.md`, `session.md`, `diff.md`, `clients.md`, `usage.md`, `ci.md`,
`github-action.md`: the reference, one page per command or area.
- `src/mcp.ts`: `toolmenu mcp`, the tools an agent calls. Its own menu must pass
toolmenu with no finding but the dry-run hint; `docs/toolmenu-badge.json` is that run.
toolmenu with no finding but the dry-run hint; `docs/toolmenu-menu.json` and
`docs/toolmenu-badge.json` are that run. CI runs the Action on it against that menu file,
and that run is the PR comment.
- `plugins/toolmenu/`: the Claude Code plugin (skills, evals, the pinned release).
- `.claude/agents/bench-finding-checker.md`: a read-only subagent that checks one bench
finding against the run's raw output.
Expand Down Expand Up @@ -111,7 +113,8 @@ A release is one commit, then a `v*` tag; the tag publishes to npm (`release.yml
pins a published release, never one ahead of npm; a test checks.

A change to `src/mcp.ts` that alters a tool's name, description or schema needs
`node dist/cli.js snapshot --no-write --badge docs/toolmenu-badge.json -- node dist/cli.js mcp`,
`node dist/cli.js snapshot --out docs/toolmenu-menu.json --badge docs/toolmenu-badge.json -- node dist/cli.js mcp`
(a test fails until it's run),
and the skills in `plugins/toolmenu/skills/` reread: they name the tools.

## Pull request descriptions
Expand Down
20 changes: 17 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Changelog

## Unreleased
## 0.28.0

### Fixes

Expand Down Expand Up @@ -29,8 +29,22 @@
`--max-reads`, the 120 s running out.
Templates and `http(s)://` URIs aren't read; a `ui://` resource a tool names stays
with `apps/resource`. `--max-reads <n>` reads n, spread over the sorted list, one
request each. No
content is kept or quoted. The reads aren't in the menu file.
request each. No content is kept or quoted, and the reads aren't in the menu file.
FINDINGS F21 has the bench run: 53 servers, 6 of them listing 13 resources a client
can't read.

### Repository

- **The PR comment checks toolmenu's own server.** CI runs the Action on `toolmenu mcp`
(6 tools) against the new `docs/toolmenu-menu.json`; the run on the three-tool fixture
still tests the Action, without a comment. A test fails when the committed menu file or
badge no longer match the server.
- **The weekly bench reads resources:** `--read-resources` on stdio servers and images,
`--max-reads 5` on hosted ones, and snapshot's outer timeout from 300 to 420 s.
`build-vendors.py` filed `lists/resource-unreadable` under `lists/resource-unread`'s
kind (it matched rule names as prefixes); an entry without a trailing `/` now matches
one rule whole.
- `docs/github-action.md`: three examples still pinned `v0.16.1`.

## 0.27.1

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ jobs:
- uses: actions/setup-node@v7
with: { node-version: 22 }
- run: npm ci && npm run build
- uses: niksa90/toolmenu@v0.27.1
- uses: niksa90/toolmenu@v0.28.0
with:
command: node dist/server.js
baseline: menu.json # your committed snapshot
Expand Down
2 changes: 1 addition & 1 deletion docs/RULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ means for semver: breaking → major, minor → minor, notice → no bump.
| `ids/authored` | info | An input that looks like an opaque ID where no tool appears to return that kind of ID, so the agent would have to invent it. Heuristic, and worded as one. Stronger when `outputSchema` is present |
| `ids/handle-lifetime` | info | 2026-07-28. A tool that creates a handle (`create`/`start`/`open`/`new`/`begin`/`acquire` + session, basket, cart, checkout, handle, lease or lock) that another tool takes back (`session_id`, `sessionToken`, `session`), whose description and output schema don't say how long it lasts. The spec asks for retention in the creation tool's description. Unsure |
| `lists/resource-read-schema` | error | `snapshot --read-resources`. Each `resources/read` result, as it came off the wire, against `ReadResourceResult` in the official schema of the negotiated protocol version: an item needs `uri` and `text` or `blob`; 2026-07-28 also requires `resultType`, `ttlMs` and `cacheScope`, earlier versions don't. One finding for the server, with the distinct schema errors, however many resources share them. A second kind: a result that matches the schema and the official SDK client still refused (seen with a `resultType` the schema takes as any string), in the SDK's words. An `InputRequiredResult` is checked against its own schema where the version has one. A version with no pinned schema isn't checked |
| `lists/resource-unread` | info | `snapshot --read-resources`. The listed resources no finding covers, by reason and with counts, so a clean result isn't read as complete: the run's failures (401/403 and other credential errors, payment, the network, a client capability, something this machine lacks, a read that didn't answer within `--timeout`), a rate limit (the reads stop at the first one), an `InputRequiredResult` (2026-07-28: the server MAY ask for input first), the 120 s for all reads running out, 64 MB of content having arrived, `--max-reads`, a listing that declares a `size` over 16 MB, `http(s)://` URIs (the spec has the client fetch those itself), a resource gone from the second `resources/list`, and a list cut at the page limit. An error-only body whose words are a credential or network failure counts here too. A missing backend counts when the server names it: "Invalid kube-config file", "You must provide an endpoint", "No currently active project" |
| `lists/resource-unread` | info | `snapshot --read-resources`. The listed resources no finding covers, by reason and with counts, so a clean result isn't read as complete: the run's failures (401/403 and other credential errors, payment, the network, a client capability, something the run lacks (a cluster, a project, an app), a read that didn't answer within `--timeout`), a rate limit (the reads stop at the first one), an `InputRequiredResult` (2026-07-28: the server MAY ask for input first), the 120 s for all reads running out, 64 MB of content having arrived, `--max-reads`, a listing that declares a `size` over 16 MB, `http(s)://` URIs (the spec has the client fetch those itself), a resource gone from the second `resources/list`, and a list cut at the page limit. An error-only body whose words are a credential or network failure counts here too. A missing backend counts when the server names it: "Invalid kube-config file", "You must provide an endpoint", "No currently active project" |
| `lists/resource-unreadable` | warn | `snapshot --lists resources --read-resources` (or `readResources` in the config) sends `resources/read` for each concrete resource `resources/list` names: once per URI, in URI order, one at a time. Resource templates aren't read, and a `ui://` resource a tool names is `apps/resource`'s. warn: the read answers with a JSON-RPC error that isn't the run's, one finding with each URI and the server's words (`-32002`, `-32602`, `ENOENT` and "not found" are the server's here: it listed the resource). No spec version says a listed resource MUST be readable, so this rests on what toolmenu saw and is never an error. warn, marked unsure: the read succeeds and every content item is text holding a JSON object with an `error` key and no key beyond `error`, `message`, `code`, `status`, `detail`, `details`, `uri` (`@dpantani/tdmcp` 0.13.2 answered `{"error": "Python class … not found."}` for five listed resources); a guess from the shape, and the body is never quoted. warn, marked unsure: two or more reads fail, all in the same words (the URI aside), none says the resource is missing, the words name no cause toolmenu knows, and nothing reads. That is a read handler that fails for everything, or one thing the run lacks that the server didn't name; the finding says both. info: an empty `contents` array, which 2026-07-28 calls ambiguous. Each resource gets one `resources/read` request: the SDK client's own retries after an `InputRequiredResult` are off for these reads. Quiet: empty text, several contents, contents with other URIs (the spec allows sub-resources), a returned `uri` or `mimeType` that differs from the listing |
| `lists/schema` | error | Each list's raw result pages against the official schema of the protocol version (`ListPromptsResult`, `ListResourcesResult`, `ListResourceTemplatesResult`); 2026-07-28 requires `ttlMs` and `cacheScope`. A declared list that fails to answer is a warning; Method not found (-32601) is info, and a setup failure (credentials, this machine) is info that says it wasn't checked. A list asked for whose capability the server doesn't declare isn't read, and the snapshot says so in a note, not a finding |
| `lists/variance` | warn | `snapshot --lists prompts,resources` (or `lists` in the config) reads `prompts/list`, `resources/list` and `resources/templates/list` (only those the server declares a capability for, up to 50 pages each, bypassing the SDK's list cache) twice on the main connection and once from a second process or connection (`--processes 1` turns that off). A difference by item (name, uri, uriTemplate), in content, key order or order is the finding |
Expand Down
2 changes: 1 addition & 1 deletion docs/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ permissions:
security-events: write # to upload SARIF
steps:
# … checkout, build, as above
- uses: niksa90/toolmenu@v0.27.1
- uses: niksa90/toolmenu@v0.28.0
with:
command: node dist/server.js
sarif: toolmenu-sarif
Expand Down
10 changes: 5 additions & 5 deletions docs/github-action.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ jobs:
- run: npm ci && npm run build
- run: npm start & # the PR's version, in the background (it keeps running for later steps)
- run: npx --yes wait-on --timeout 60000 tcp:localhost:3000 # fail after 60 s instead of hanging
- uses: niksa90/toolmenu@v0.16.1
- uses: niksa90/toolmenu@v0.28.0
with:
url: http://localhost:3000/mcp
headers: "x-mcp-api-key: ${{ env.MCP_API_KEY }}"
Expand All @@ -111,7 +111,7 @@ tools behind an unlock: a renamed, removed or no-longer-read-only one never reac
the report. Let the scenario build the baseline instead:

```yaml
- uses: niksa90/toolmenu@v0.16.1
- uses: niksa90/toolmenu@v0.28.0
with:
command: node dist/server.js
scenario: scenario.yml # unlocks each toolset, lists after each
Expand Down Expand Up @@ -174,7 +174,7 @@ check even though nothing was checked. The PR comment and a notice say so, and t
`skipped` output is `'true'`. To make a skip fail instead, gate on it:

```yaml
- uses: niksa90/toolmenu@v0.16.1
- uses: niksa90/toolmenu@v0.28.0
id: toolmenu
with:
url: https://mcp.example.com/mcp # a remote server that needs a key
Expand Down Expand Up @@ -204,7 +204,7 @@ that adds findings at or above `fail-on` fails even when nothing breaks. Snapsho
session findings count either way.

```yaml
- uses: niksa90/toolmenu@v0.27.1
- uses: niksa90/toolmenu@v0.28.0
with:
command: node dist/server.js
gate: review
Expand Down Expand Up @@ -247,7 +247,7 @@ version in a comment, and let Dependabot's `github-actions` updates keep the pin
current:

```yaml
- uses: niksa90/toolmenu@<commit SHA of the release> # v0.27.1
- uses: niksa90/toolmenu@<commit SHA of the release> # v0.28.0
```

`toolmenu init` writes the actions it sets up (checkout, setup-node, …) pinned this
Expand Down
Loading
Loading