Skip to content

0.29.0: instructions/unknown-name, spec/discover-refused, config rule-id warning, auth login and diff notes - #83

Open
niksa90 wants to merge 10 commits into
mainfrom
claude/loving-ritchie-4clta3
Open

niksa90 wants to merge 10 commits into
mainfrom
claude/loving-ritchie-4clta3

Conversation

@niksa90

@niksa90 niksa90 commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

What

  • instructions/unknown-name (warn, new): the handshake's instructions name a tool the menu doesn't list, or, with --lists resources, a resource URI in the server's own scheme that no listed resource or template matches (RFC 6570 operators included). Tool names are read from the words, so those findings are marked unsure. Left out: fields (open_url, read_only, execute_budget.packs), values (action=update_budget, stop_loss: 2), subcommands of a listed tool, listed prompts, namespaced tool names, and names in a sentence that says why they're missing (old names; a token, API key, account or login next to "only", "needs", "adds", "with an"; an unlock). A server that declares prompts whose list wasn't read gets a note that a name may be a prompt. A resources list cut at the page limit isn't checked against. The instructions are now kept on the connection (src/connect.ts) and passed to the rules.
  • spec/discover-refused (warn, new): server/discover got 401 or 403 while the 2025 handshake served the menu. 0.28.1 falls back past it; this tells the server's author, since a client that tries 2026-07-28 first stops there. spec/discover isn't said as well.
  • Config: a rules id that isn't a rule gets a stderr line with the closest ids (Did you mean naming/vague-id?) instead of being ignored in silence.
  • auth login: when the server didn't challenge during the login and a token from an earlier one was sent, it says so, with how many tools the server lists without a login and the commands to compare or log out.
  • diff/server-version-stale (info, new): with --release, the server reported the same serverInfo.version in both releases.
  • Plugin: plugins/toolmenu on toolmenu 0.28.1 (package.json, .claude-plugin/plugin.json, lockfile: only the toolmenu entry and its integrity hash change).
  • Bench: bench/results/2026-11-19-instructions/ and FINDINGS F22.
  • Release: 0.29.0 in package.json, package-lock.json, server.json, the CHANGELOG, and the Action pins in README.md, docs/github-action.md and docs/ci.md. With the retirement of description/late-instruction (Retire description/late-instruction #82).

Why

Dreamwork 1.10.0's instructions told the agent to call submit_feedback while its menu didn't have it; the tool appeared a day later under the same version. lahend.ee names get_ruling, mcp.coworkingview.com names get_best_coworking, pageaudit.online names create_tab: none is in its menu. Dreamwork, api.getopservo.com, patlas.dev and clearlist.me answer server/discover with 401 and serve their menu over 2025. A misspelled rule id in the config turned nothing up, so a CI gate on it wasn't one. serbia-prices-mcp 0.1.5 reports FastMCP's own 4.2.1 as its version.

Tests

  • npm test: 734 passing.
  • instructions/unknown-name through the CLI on a fixture server: missing tools and a missing resource, the order, the quoted sentence (the mention that counted, not an earlier mode= value), the message, detail and next step, the prompts note; quiet on templates with operators, a bold URI, a <slug> placeholder, listed prompts and each kind of noise above, and still firing when "token" appears in an unrelated sentence.
  • spec/discover-refused through the CLI for 401 and 403, checked against the SDK client 2.4: with versionNegotiation: auto it fails with "Version negotiation failed", on its default (legacy) it connects. Quiet on a 2025 server and with --protocol legacy.
  • Bench, 2026-11-19 (bench/results/2026-11-19-instructions/, FINDINGS F22): 1,078 hosted servers listed without credentials. instructions/unknown-name fired on 52 before the filters and 16 after; each of the 16 was read. One is a tool on the same vendor's other server (marked unsure). spec/discover-refused fires on the 3 of 733 login-walled servers the fallback reached, and on none of the 1,078.
  • Config warning, auth login with a stored token, and diff/server-version-stale (through toolmenu diff --release and through diffMenus, including a server that reports its release): each has its own test.
  • Plugin: node --test test/mcp.test.mjs, including the check that the plugin pins a version npm has.

plugins/toolmenu/package.json and .claude-plugin/plugin.json to 0.28.1, now
on npm, and its lockfile (npm install --package-lock-only --ignore-scripts):
only the toolmenu entry changes, with npm's integrity hash.
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

toolmenu

Verdict: ✓ unchanged: the same menu and the same findings. Nothing to accept. Findings against the baseline: 0 new, 0 gone, 0 changed.

toolmenu diff: toolmenu 0.28.0 → 0.29.0 (server-reported)

No breaking changes · 0 minor · 0 notices · ±0 tokens (~1,523 → ~1,523, estimate) · 6 → 6 tools

Version: suggested bump: none · not checked (set the Action's release input old..new)

No findings.

✓ 0 errors, 0 warnings, 0 info

Baseline: docs/toolmenu-menu.json as of main

toolmenu snapshot: toolmenu 0.29.0

Protocol 2026-07-28 · 6 tools · ~1,523 tokens (estimate)

1 info
Rule Finding
info write/no-dry-run · unsure 2 tools are annotated as destructive, and toolmenu found no dry_run/preview-style parameter and no preview tool for the same thing. Where a mistake is costly to undo, a dry run lets the agent see what would change while it can still back out. · snapshot_menu, check_session
→ Next: Where the effect is hard to undo, add a dry_run parameter that returns what would change without changing it. If it's cheap to undo, or a preview exists under another name, ignore this.

0 errors, 0 warnings, 1 info

Where the tokens go (estimate): the top 5 tools are 93% of the menu
Share Tool Tokens Description Schema
30% snapshot_menu ~461 ~99 ~353
26% check_session ~394 ~102 ~283
15% review_baseline ~231 ~65 ~156
12% diff_menus ~187 ~39 ~138
10% check_auth ~149 ~86 ~54
  • repeated: cwd (~37 tokens) in 4 tools, ~148 in all

A tool's schema can't $ref another tool's, so a block shared across tools is paid once per tool: only a smaller block saves tokens there.

toolmenu · token counts are estimates

A typo like "naming/vauge-id": "error" was ignored without a word, so a CI
gate on it wasn't one. The config loader now writes a line to stderr with
the closest ids (the same lookup explain uses) and runs.
A token stored earlier, sent to a server that never challenged, read as
"Logged in" with nothing else. When no 401 came during the login,
toolmenu lists once without credentials and says how many tools the
server serves that way, with the commands to compare or log out.
With --release, an info note when serverInfo.version is the same in both
menus while the release changed: whatever tracks the server by that
version sees no change (serbia-prices-mcp 0.1.5 reports FastMCP's 4.2.1).
The $ref-refactor test's after-menu now reports its release.
…sn't list

The handshake's instructions are kept on the connection and passed to the
rules. A snake_case name that reads as a tool, or a resource URI in the
server's own scheme, that the menu doesn't list is a warning, marked
unsure for tool names. Fields, values, subcommands, prompts and names in
a sentence that says why they're missing are left out: on 1,073 hosted
servers listed without credentials, that took the count from 52 to 15.
Dreamwork 1.10.0 named submit_feedback a day before its menu had it.
@niksa90 niksa90 changed the title The plugin on toolmenu 0.28.1 instructions/unknown-name, config rule-id warning, auth login and diff notes; the plugin on 0.28.1 Nov 19, 2026
- Resource URIs: a Markdown-bold URI is read without its asterisks; RFC
  6570 operators in a template ({+path}, {?q}, {/seg}) match what they
  expand to; a resources list cut at the page limit isn't checked against.
- The detail quotes the mention that counted, found with the same word
  boundaries, and names are in the order those mentions come.
- A sentence explains a missing tool only when a condition word (token,
  account, login, scope) sits near an availability word (only, needs,
  adds, unlock): "token usage" alone no longer hides a missing tool.
- A server that declares prompts whose list wasn't read: the finding says
  a name may be a prompt and to list them with --lists prompts.
- diff/server-version-stale: a CLI test in diff.test.mjs, and the case of
  a server that reports its release; its row in RULES.md in order.
- auth.ts: check() has its doc comment back.
The full bench pass after the narrower explanation check found three
servers it missed: "With an API key (Bearer rs_live_...)" and "A free
account at https://.../account adds" explain why tools are missing, and
execute_budget.packs is a field path. Keys count as a condition, a full
stop inside a URL doesn't end the sentence, and a name followed by .word
is a path. On 1,078 hosted servers listed without credentials, 16 get it.
…F22)

instructions/unknown-name over the 1,120 open hosted servers of the
2026-11-09 corpus: 1,078 listed, 16 get the finding, 52 before its
filters. The 733 that needed a login on 2026-11-09, listed again with
0.28.1: 5 list now, 3 of them through the server/discover fallback.
…ke works

When server/discover gets 401 or 403 and the 2025 initialize serves the
menu, toolmenu falls back (0.28.1) and now says so to the server's author:
a client that tries 2026-07-28 first stops there. Checked in the tests
against the SDK client 2.4: with versionNegotiation auto it fails with
"Version negotiation failed"; its default (legacy) connects. spec/discover
isn't said as well. Fires on Dreamwork 1.10.0 and the 3 hosted servers the
fallback reached in F22; on the 1,078 open ones, on none.
@niksa90 niksa90 changed the title instructions/unknown-name, config rule-id warning, auth login and diff notes; the plugin on 0.28.1 0.29.0: instructions/unknown-name, spec/discover-refused, config rule-id warning, auth login and diff notes Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants