Repository navigation
Conversation
plugins/toolmenu/package.json and .claude-plugin/plugin.json to 0.28.1, now on npm, and its lockfile (npm install --package-lock-only --ignore-scripts): only the toolmenu entry changes, with npm's integrity hash.
toolmenuVerdict: ✓ unchanged: the same menu and the same findings. Nothing to accept. Findings against the baseline: 0 new, 0 gone, 0 changed. toolmenu diff:
|
| Rule | Finding | |
|---|---|---|
| info | write/no-dry-run · unsure |
2 tools are annotated as destructive, and toolmenu found no dry_run/preview-style parameter and no preview tool for the same thing. Where a mistake is costly to undo, a dry run lets the agent see what would change while it can still back out. · snapshot_menu, check_session → Next: Where the effect is hard to undo, add a dry_run parameter that returns what would change without changing it. If it's cheap to undo, or a preview exists under another name, ignore this. |
0 errors, 0 warnings, 1 info
Where the tokens go (estimate): the top 5 tools are 93% of the menu
| Share | Tool | Tokens | Description | Schema |
|---|---|---|---|---|
| 30% | snapshot_menu |
~461 | ~99 | ~353 |
| 26% | check_session |
~394 | ~102 | ~283 |
| 15% | review_baseline |
~231 | ~65 | ~156 |
| 12% | diff_menus |
~187 | ~39 | ~138 |
| 10% | check_auth |
~149 | ~86 | ~54 |
- repeated: cwd (~37 tokens) in 4 tools, ~148 in all
A tool's schema can't $ref another tool's, so a block shared across tools is paid once per tool: only a smaller block saves tokens there.
toolmenu · token counts are estimates
A typo like "naming/vauge-id": "error" was ignored without a word, so a CI gate on it wasn't one. The config loader now writes a line to stderr with the closest ids (the same lookup explain uses) and runs.
A token stored earlier, sent to a server that never challenged, read as "Logged in" with nothing else. When no 401 came during the login, toolmenu lists once without credentials and says how many tools the server serves that way, with the commands to compare or log out.
With --release, an info note when serverInfo.version is the same in both menus while the release changed: whatever tracks the server by that version sees no change (serbia-prices-mcp 0.1.5 reports FastMCP's 4.2.1). The $ref-refactor test's after-menu now reports its release.
…sn't list The handshake's instructions are kept on the connection and passed to the rules. A snake_case name that reads as a tool, or a resource URI in the server's own scheme, that the menu doesn't list is a warning, marked unsure for tool names. Fields, values, subcommands, prompts and names in a sentence that says why they're missing are left out: on 1,073 hosted servers listed without credentials, that took the count from 52 to 15. Dreamwork 1.10.0 named submit_feedback a day before its menu had it.
- Resource URIs: a Markdown-bold URI is read without its asterisks; RFC
6570 operators in a template ({+path}, {?q}, {/seg}) match what they
expand to; a resources list cut at the page limit isn't checked against.
- The detail quotes the mention that counted, found with the same word
boundaries, and names are in the order those mentions come.
- A sentence explains a missing tool only when a condition word (token,
account, login, scope) sits near an availability word (only, needs,
adds, unlock): "token usage" alone no longer hides a missing tool.
- A server that declares prompts whose list wasn't read: the finding says
a name may be a prompt and to list them with --lists prompts.
- diff/server-version-stale: a CLI test in diff.test.mjs, and the case of
a server that reports its release; its row in RULES.md in order.
- auth.ts: check() has its doc comment back.
The full bench pass after the narrower explanation check found three servers it missed: "With an API key (Bearer rs_live_...)" and "A free account at https://.../account adds" explain why tools are missing, and execute_budget.packs is a field path. Keys count as a condition, a full stop inside a URL doesn't end the sentence, and a name followed by .word is a path. On 1,078 hosted servers listed without credentials, 16 get it.
…F22) instructions/unknown-name over the 1,120 open hosted servers of the 2026-11-09 corpus: 1,078 listed, 16 get the finding, 52 before its filters. The 733 that needed a login on 2026-11-09, listed again with 0.28.1: 5 list now, 3 of them through the server/discover fallback.
…ke works When server/discover gets 401 or 403 and the 2025 initialize serves the menu, toolmenu falls back (0.28.1) and now says so to the server's author: a client that tries 2026-07-28 first stops there. Checked in the tests against the SDK client 2.4: with versionNegotiation auto it fails with "Version negotiation failed"; its default (legacy) connects. spec/discover isn't said as well. Fires on Dreamwork 1.10.0 and the 3 hosted servers the fallback reached in F22; on the 1,078 open ones, on none.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
instructions/unknown-name(warn, new): the handshake'sinstructionsname a tool the menu doesn't list, or, with--lists resources, a resource URI in the server's own scheme that no listed resource or template matches (RFC 6570 operators included). Tool names are read from the words, so those findings are marked unsure. Left out: fields (open_url,read_only,execute_budget.packs), values (action=update_budget,stop_loss: 2), subcommands of a listed tool, listed prompts, namespaced tool names, and names in a sentence that says why they're missing (old names; a token, API key, account or login next to "only", "needs", "adds", "with an"; an unlock). A server that declares prompts whose list wasn't read gets a note that a name may be a prompt. A resources list cut at the page limit isn't checked against. The instructions are now kept on the connection (src/connect.ts) and passed to the rules.spec/discover-refused(warn, new):server/discovergot 401 or 403 while the 2025 handshake served the menu. 0.28.1 falls back past it; this tells the server's author, since a client that tries 2026-07-28 first stops there.spec/discoverisn't said as well.rulesid that isn't a rule gets a stderr line with the closest ids (Did you mean naming/vague-id?) instead of being ignored in silence.auth login: when the server didn't challenge during the login and a token from an earlier one was sent, it says so, with how many tools the server lists without a login and the commands to compare or log out.diff/server-version-stale(info, new): with--release, the server reported the sameserverInfo.versionin both releases.plugins/toolmenuon toolmenu 0.28.1 (package.json,.claude-plugin/plugin.json, lockfile: only thetoolmenuentry and its integrity hash change).bench/results/2026-11-19-instructions/and FINDINGS F22.package.json,package-lock.json,server.json, the CHANGELOG, and the Action pins inREADME.md,docs/github-action.mdanddocs/ci.md. With the retirement ofdescription/late-instruction(Retire description/late-instruction #82).Why
Dreamwork 1.10.0's instructions told the agent to call
submit_feedbackwhile its menu didn't have it; the tool appeared a day later under the same version.lahend.eenamesget_ruling,mcp.coworkingview.comnamesget_best_coworking,pageaudit.onlinenamescreate_tab: none is in its menu. Dreamwork,api.getopservo.com,patlas.devandclearlist.meanswerserver/discoverwith 401 and serve their menu over 2025. A misspelled rule id in the config turned nothing up, so a CI gate on it wasn't one.serbia-prices-mcp0.1.5 reports FastMCP's own 4.2.1 as its version.Tests
npm test: 734 passing.instructions/unknown-namethrough the CLI on a fixture server: missing tools and a missing resource, the order, the quoted sentence (the mention that counted, not an earliermode=value), the message, detail and next step, the prompts note; quiet on templates with operators, a bold URI, a<slug>placeholder, listed prompts and each kind of noise above, and still firing when "token" appears in an unrelated sentence.spec/discover-refusedthrough the CLI for 401 and 403, checked against the SDK client 2.4: withversionNegotiation: autoit fails with "Version negotiation failed", on its default (legacy) it connects. Quiet on a 2025 server and with--protocol legacy.bench/results/2026-11-19-instructions/, FINDINGS F22): 1,078 hosted servers listed without credentials.instructions/unknown-namefired on 52 before the filters and 16 after; each of the 16 was read. One is a tool on the same vendor's other server (marked unsure).spec/discover-refusedfires on the 3 of 733 login-walled servers the fallback reached, and on none of the 1,078.auth loginwith a stored token, anddiff/server-version-stale(throughtoolmenu diff --releaseand throughdiffMenus, including a server that reports its release): each has its own test.node --test test/mcp.test.mjs, including the check that the plugin pins a version npm has.