Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,28 @@

## Unreleased

## 0.29.0

### New

- **`instructions/unknown-name`** (warn): the server's `instructions` name a tool the
menu doesn't list, or, with `--lists resources`, a resource URI no listed resource or
template matches. Dreamwork 1.10.0's instructions told the agent to call
`submit_feedback` while its menu didn't have it (it was added a day later, under the
same version). Tool names are read from the words, so the finding is marked unsure;
fields, values, subcommands, prompts and names in a sentence that says why they're
missing (old names, a login, a token, an API key) are left out. On 1,078 hosted
servers listed without credentials, 16 get it, from 52 before those filters.

- **`spec/discover-refused`** (warn): `server/discover` refused with 401 or 403 while the
2025 handshake serves the menu. 0.28.1 falls back past it; this says so to the server's
author, since a client that tries 2026-07-28 first stops there (the official SDK client
with `versionNegotiation: auto`). Dreamwork 1.10.0 and 3 of 733 hosted servers that
needed a login on 2026-11-09 do this.
- **`diff/server-version-stale`** (info, with `--release`): the server reported the
same `serverInfo.version` in both releases, so whatever tracks it by that version
sees no change. `serbia-prices-mcp` 0.1.5 reports FastMCP's own 4.2.1.

### Changes

- **`description/late-instruction` is retired.** It flagged instructions past
Expand All @@ -11,6 +33,20 @@
`descriptionLimit`. A config that still names the rule loads, with a line saying it
changes nothing; `toolmenu explain description/late-instruction` says why it went.

- **A config rule id that isn't a rule is no longer ignored in silence.**
`"naming/vauge-id": "error"` used to change nothing without a word, so a CI gate on
it wasn't one. toolmenu now says so on stderr, with the closest ids, and runs.

- **`auth login` says when the server didn't ask for the login.** A token stored
earlier, sent to a server that never challenged, read as "Logged in" with nothing
else; it now adds how many tools the server lists without a login, and the commands
to compare or log out.

### Repository

- **The plugin runs toolmenu 0.28.1** (was 0.28.0): `snapshot` lists a server that
refuses `server/discover` but serves a menu over the 2025 handshake.

## 0.28.1

### Fixes
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ jobs:
- uses: actions/setup-node@v7
with: { node-version: 22 }
- run: npm ci && npm run build
- uses: niksa90/toolmenu@v0.28.1
- uses: niksa90/toolmenu@v0.29.0
with:
command: node dist/server.js
baseline: menu.json # your committed snapshot
Expand Down
31 changes: 31 additions & 0 deletions bench/results/2026-11-19-instructions/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Bench run, 2026-11-19: server instructions, and the 0.28.1 handshake fallback

Two passes over the hosted (Streamable HTTP) servers of the 2026-11-09 run
(`../weekly/2026-11-09/http/servers.tsv`), each listing only: no tool was called, no
credentials were sent (`--no-auth`), one connection per server (`--processes 1`).

## `instructions/unknown-name` (FINDINGS F22)

- **The servers:** the 1,120 marked `open` on 2026-11-09. 1,078 listed tools this time.
- **Run with:** `snapshot --json --lists resources --processes 1 --no-auth --timeout 30000`,
at commit `5c5f393` for every server, then the 19 it fired on again at `08547c2` (that
commit only removes findings: an API key or account in the sentence, a full stop inside
a URL, a dotted field path).
- **`servers.tsv`:** one line per server. `menu` is `listed` or `no-menu`; `unknown_names`
the names the rule reports for it, at `08547c2`.
- **`findings.jsonl`:** every `instructions/unknown-name` finding at `08547c2`: 16 servers.

The first pass of the rule, before it left out fields, values, subcommands, prompts and
names in a sentence that explains them, fired on 52 of these servers.

## The handshake fallback (0.28.1)

- **The servers:** the 733 marked `oauth` with no tools on 2026-11-09.
- **Run with:** `snapshot --json --processes 1 --no-auth --timeout 30000`, built from
`62d1a48` (0.28.1, whose fallback gives a `server/discover` refused with 401 or 403 a 2025
`initialize`). Partway through, the build was replaced by one that only added rules; how
a server is connected to and listed didn't change.
- **`login-rerun.tsv`:** `menu_2026-11-19` is `listed` or `no-menu`. For the five that list,
`server_discover` is the HTTP status a `server/discover` request got, sent by hand the
same day: 401 for the three the fallback reaches, 400 for two that answer it as 2025
servers do, which 0.28.0 already handled.
16 changes: 16 additions & 0 deletions bench/results/2026-11-19-instructions/findings.jsonl

Large diffs are not rendered by default.

Loading
Loading