Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@

## Unreleased

### Changes

- **`description/late-instruction` is retired.** It flagged instructions past
character 280, a cut that no client documents: the number came from one private
client. Claude Code documents 16,384 and 4,096; `description/buried` checks those,
the `amazon-q` preset's 10,024 (reported, not documented), or your own
`descriptionLimit`. A config that still names the rule loads, with a line saying it
changes nothing; `toolmenu explain description/late-instruction` says why it went.

## 0.28.1

### Fixes
Expand Down
10 changes: 5 additions & 5 deletions docs/PROTOCOLS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ exactly the one in the row.

| Protocol | Handshake | `tools/list` against its schema | Mid-session change, pinned to its step | A change only this session sees | `snapshot` rules that apply |
|---|---|---|---|---|---|
| 2024-11-05 | `initialize` | ✓ | ✓ `session/mid-insert` (error), `list_changed` received | allowed: no finding | 41 of 47 |
| 2025-03-26 | `initialize` | ✓ | ✓ `session/mid-insert` (error), `list_changed` received | allowed: no finding | 41 of 47 |
| 2025-06-18 | `initialize` | ✓ | ✓ `session/mid-insert` (error), `list_changed` received | allowed: no finding | 41 of 47 |
| 2025-11-25 | `initialize` | ✓ | ✓ `session/mid-insert` (error), `list_changed` received | allowed: no finding | 41 of 47 |
| 2026-07-28 | `server/discover` | ✓ | ✓ `session/mid-insert` (error), `list_changed` received | `session/side-effect` (warn) | 47 of 47 |
| 2024-11-05 | `initialize` | ✓ | ✓ `session/mid-insert` (error), `list_changed` received | allowed: no finding | 40 of 46 |
| 2025-03-26 | `initialize` | ✓ | ✓ `session/mid-insert` (error), `list_changed` received | allowed: no finding | 40 of 46 |
| 2025-06-18 | `initialize` | ✓ | ✓ `session/mid-insert` (error), `list_changed` received | allowed: no finding | 40 of 46 |
| 2025-11-25 | `initialize` | ✓ | ✓ `session/mid-insert` (error), `list_changed` received | allowed: no finding | 40 of 46 |
| 2026-07-28 | `server/discover` | ✓ | ✓ `session/mid-insert` (error), `list_changed` received | `session/side-effect` (warn) | 46 of 46 |

- **Handshake**: the request toolmenu connected with, and the server answering that version.
- **`tools/list` against its schema**: `spec/schema` passes a valid result and reports an
Expand Down
1 change: 0 additions & 1 deletion docs/RULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ means for semver: breaking → major, minor → minor, notice → no bump.
| `catalog/skipped` | info | `--catalog`: tools that look like command routers but weren't called, because no listing mode needs no command. toolmenu never sends a command to list one |
| `description/buried` | warn | A sentence that reads as an instruction to the agent sits past the point where the client cuts descriptions, so the model never sees it. Read from the words, so marked unsure |
| `description/cut` | info | Descriptions longer than the client sends, with where the cut falls. Nothing past it reads as an instruction, so it's a check, not a bug |
| `description/late-instruction` | info | Only with the default cut: an instruction past 280 characters, where one real client cut, but inside the default cut. Split out of `description/cut` in 0.13; a setting for `description/cut` still applies to it unless it's set itself (`RULE_ALIASES`) |
| `ids/authored` | info | An input that looks like an opaque ID where no tool appears to return that kind of ID, so the agent would have to invent it. Heuristic, and worded as one. Stronger when `outputSchema` is present |
| `ids/handle-lifetime` | info | 2026-07-28. A tool that creates a handle (`create`/`start`/`open`/`new`/`begin`/`acquire` + session, basket, cart, checkout, handle, lease or lock) that another tool takes back (`session_id`, `sessionToken`, `session`), whose description and output schema don't say how long it lasts. The spec asks for retention in the creation tool's description. Unsure |
| `lists/resource-read-schema` | error | `snapshot --read-resources`. Each `resources/read` result, as it came off the wire, against `ReadResourceResult` in the official schema of the negotiated protocol version: an item needs `uri` and `text` or `blob`; 2026-07-28 also requires `resultType`, `ttlMs` and `cacheScope`, earlier versions don't. One finding for the server, with the distinct schema errors, however many resources share them. A second kind: a result that matches the schema and the official SDK client still refused (seen with a `resultType` the schema takes as any string), in the SDK's words. An `InputRequiredResult` is checked against its own schema where the version has one. A version with no pinned schema isn't checked |
Expand Down
4 changes: 4 additions & 0 deletions docs/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,10 @@ Keyword matching only in v1: deterministic, free and reproducible in CI.

**Spec** (not observed failures: protocol requirements, kept minimal)

> **Note, 2026-11-18 (after the freeze).** `description/late-instruction` is retired:
> 280 was one private client's cut, and no client documents it. `description/buried`
> checks the documented cuts, or the `descriptionLimit` a user sets.

| id | sev | check |
|---|---|---|
| `spec/x-mcp-header` | error / warn | `since: 2026-07-28`. An `x-mcp-header` annotation the spec doesn't allow (SEP-2243): empty; not an RFC 9110 token; the same header twice ignoring case; on a parameter that isn't `string`, `integer` or `boolean`, or has no single `type`; anywhere but a parameter reached through `properties` alone. Streamable HTTP clients **MUST** leave such a tool out of `tools/list`, and the official SDK client does, with only a console warning. Error on an HTTP server, warn on stdio (clients there MAY ignore the annotation). A `number` parameter counts too: the SDK client drops those tools since 2.4. Every broken annotation in a tool is listed (the SDK stops at the first). Checked against the SDK client's behaviour in the tests |
Expand Down
3 changes: 1 addition & 2 deletions docs/snapshot.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ server, or from the spec itself. If a rule can't point to one, it doesn't ship.
| `naming/route` | error | A `routes.yml` expectation broke: a keyword now matches the wrong tool at least as well as the right one |
| `description/buried` | warn | Instructions to the agent ("use X instead", "don't retry", "never guess one") past the point where your client cuts descriptions: in Claude Code 16,384 characters for a tool loaded through tool search (its default) and 4,096 for one that loads up front, or your `descriptionLimit`. From a real failure: a client cut at 280 characters, and the line that decided routing was at 1,222 |
| `description/cut` | info | Descriptions longer than the client sends, with nothing that reads as an instruction past the cut, as one summary that shows where each one is cut. The model gets a prefix that can read as complete |
| `description/late-instruction` | info | With the default cut: instructions after character 280, which a client that cuts descriptions short would hide, as one summary. Set `descriptionLimit` and `description/buried` checks your real cut instead. (Before 0.13 this was part of `description/cut`; that id's setting still applies.) |
| `naming/vague-id` | warn | Parameters called just `id` that don't say *which* thing, as one summary with a suggested name |
| `ids/authored` | info | An ID the agent must supply that no tool appears to return, so the agent may invent it, one finding per kind of ID (heuristic: IDs from a URL or another tool's text output look the same) |
| `ids/handle-lifetime` | info | 2026-07-28: a tool that creates a handle another tool takes back (`start_session` → `session_id`) doesn't say how long it lasts (heuristic) |
Expand Down Expand Up @@ -163,7 +162,7 @@ Claude Code with tool search off) or
`"amazon-q"` (10,024, reported in
[makenotion/notion-mcp-server#145](https://github.com/makenotion/notion-mcp-server/issues/145)).
Clients that send descriptions in full don't need a setting beyond that. If your own
client cuts shorter, as mine did at 280, set its number. `fullDescriptions` lists the
client cuts shorter, set its number. `fullDescriptions` lists the
tools it sends uncut, so the description rules skip them.

## Every server you use: `scan`
Expand Down
6 changes: 6 additions & 0 deletions src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import { CLI_OPTIONS } from './options.js';
import { editDistance } from './words.js';
import { formatScan, ListError, loadList, scan, scanExit, type AllowEnv } from './scan.js';
import { explain, findRule, formatExplained, listRules, ruleDocs } from './rule-docs.js';
import { RETIRED_RULES } from './rules/index.js';
import { commandHelp, commandOptions, isCommand, overview, unknownCommand, type Command } from './help.js';

class UsageError extends Error {}
Expand Down Expand Up @@ -593,6 +594,11 @@ function explainCommand(rest: string[], command: string[] | undefined, format: F
process.stdout.write(format === 'json' ? `${JSON.stringify(Object.keys(ruleDocs()))}\n` : listRules());
return 0;
}
const retired = rest[0].trim().toLowerCase();
if (Object.hasOwn(RETIRED_RULES, retired)) {
process.stdout.write(format === 'json' ? `${JSON.stringify({ id: retired, retired: RETIRED_RULES[retired] })}\n` : `${retired}: ${RETIRED_RULES[retired]}\n`);
return 0;
}
const found = findRule(rest[0]);
if ('near' in found) {
const hint = found.near.length ? ` Did you mean ${found.near.join(', ')}?` : '';
Expand Down
2 changes: 2 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { readFile } from 'node:fs/promises';
import { CLIENTS } from './clients.js';
import { LIST_WORDS } from './lists.js';
import { CLIENT_LIMITS } from './rules/description.js';
import { RETIRED_RULES } from './rules/index.js';
import type { Severity } from './types.js';
import { jsonErrorAt } from './menu.js';

Expand Down Expand Up @@ -70,6 +71,7 @@ export async function loadConfig(path?: string): Promise<Config> {
if (!['error', 'warn', 'info', 'off'].includes(value)) {
throw new Error(`${file}: rules["${id}"] must be error, warn, info or off`);
}
if (Object.hasOwn(RETIRED_RULES, id)) process.stderr.write(`toolmenu: ${file}: rules["${id}"] changes nothing: ${RETIRED_RULES[id]}\n`);
}
if (config.protocol !== undefined && !['auto', 'legacy', 'modern'].includes(config.protocol)) {
throw new Error(`${file}: protocol must be auto, legacy or modern`);
Expand Down
38 changes: 0 additions & 38 deletions src/rules/description.ts
Original file line number Diff line number Diff line change
Expand Up @@ -152,9 +152,6 @@ export const buried: Rule = {
},
};

/** A short cut to check against when the client's own cut isn't set: one real client cut at 280. */
const SHORT_CUT = 280;

/** “…the last words before ✂ the first words after…”, to find the cut in the text. */
function aroundTheCut(text: string, at: number): string {
const before = text.slice(Math.max(0, at - 30), at).replace(/\s+/g, ' ');
Expand Down Expand Up @@ -197,38 +194,3 @@ export const cut: Rule = {
},
};

export const lateInstruction: Rule = {
id: 'description/late-instruction',
scope: 'agent',
severity: 'info',
lesson: 'truncated description',
summary: 'Instructions to the agent that a client cutting descriptions short would hide',
run(ctx) {
// Only with the default cut: once descriptionLimit is set, description/buried checks the real one.
if (ctx.descriptionLimit !== undefined) return [];
const { limit, who } = resolveLimit(undefined);
if (limit <= SHORT_CUT) return [];
const all = ctx.menu.tools.map((t) => t.name);
const late: { tool: MenuTool; at: number; straddles: boolean; sentence: string; more: number }[] = [];
for (const tool of kept(ctx.menu.tools, ctx)) {
const text = tool.description ?? '';
if (text.length <= SHORT_CUT || sentInFull(tool, ctx.fullDescriptions)) continue;
// Past the default cut is description/buried's.
const hidden = pastTheCut(text, all.filter((n) => n !== tool.name), SHORT_CUT).filter((i) => i.at < limit);
if (hidden.length) late.push({ tool, at: hidden[0].at, straddles: hidden[0].straddles, sentence: sentenceAround(text, hidden[0].at), more: hidden.length - 1 });
}
if (late.length === 0) return [];
const one = late.length === 1;
const client = who;
return [
{
...(one ? { tool: late[0].tool.name } : {}),
confidence: 'unsure',
message: `${one ? `The ${late[0].tool.name} description gives` : `${late.length} descriptions give`} the agent instructions ${late.every((l) => l.straddles) ? `that run past character ${SHORT_CUT}` : late.some((l) => l.straddles) ? `after character ${SHORT_CUT} or running past it` : `after character ${SHORT_CUT}`}, where a client that cuts descriptions short never shows them (one real client cut at ${SHORT_CUT}). ${client} sends ${num(limit)} characters, so there they arrive.`,
// One that starts before the cut and runs past it is cut in the middle: say so, or "char 264" reads as before the cut.
detail: late.map((l) => `${l.tool.name}: char ${num(l.at)}${l.straddles ? ` (runs past ${SHORT_CUT})` : ''}: “${clip(l.sentence, 120)}”${l.more ? ` (and ${l.more} more after it)` : ''}`),
fix: `If your client cuts descriptions, set descriptionLimit to its cut and toolmenu checks against that. Otherwise, moving ${one ? 'the instruction' : 'each instruction'} into the first sentences is cheap insurance.`,
},
];
},
};
15 changes: 8 additions & 7 deletions src/rules/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { Finding, Severity } from '../types.js';
import { SEVERITY_RANK } from '../types.js';
import { buried, cut, lateInstruction } from './description.js';
import { buried, cut } from './description.js';
import { connectionVariance, hostSpecific, nondeterministic, processVariance } from './determinism.js';
import { authoredIds } from './ids.js';
import { clientFeatures, clientLimit, clientName } from './clients.js';
Expand Down Expand Up @@ -43,7 +43,6 @@ export const MENU_RULES: Rule[] = [
handleLifetime,
buried,
cut,
lateInstruction,
noDryRun,
unannotated,
readOnlyMismatch,
Expand Down Expand Up @@ -75,11 +74,13 @@ export const MENU_RULES: Rule[] = [
];

/**
* Rules split out of an older one. A setting for the old id still applies to the
* new one unless the new id is set itself: before 0.13, description/cut held
* the late-instruction hint too, and `"description/cut": "off"` turned both off.
* Rules that no longer exist, and why. A config may still name one: it changes nothing,
* and loading the config says so. `explain` says the same.
*/
export const RULE_ALIASES: Record<string, string> = { 'description/late-instruction': 'description/cut' };
export const RETIRED_RULES: Record<string, string> = {
'description/late-instruction':
"retired after 0.28.1: it checked a cut at 280 characters, which no client documents. If your client cuts descriptions, set descriptionLimit to its cut and description/buried checks against it.",
};

export interface RuleSettings {
/** Per-rule severity override, or 'off'. */
Expand All @@ -95,7 +96,7 @@ export function runRules(rules: Rule[], ctx: RuleContext, settings: RuleSettings
// The menu as an MCP Apps host shows it to the agent: app-only tools left out.
const shown = agentMenu(ctx.menu).shown;
for (const rule of rules) {
const configured = settings.rules?.[rule.id] ?? (RULE_ALIASES[rule.id] ? settings.rules?.[RULE_ALIASES[rule.id]] : undefined);
const configured = settings.rules?.[rule.id];
if (configured === 'off') continue;
if (rule.since && (!ctx.protocolVersion || ctx.protocolVersion < rule.since)) continue;
for (const f of rule.run({ ...ctx, ...(rule.scope === 'agent' ? { menu: shown } : {}), isIgnored })) {
Expand Down
9 changes: 9 additions & 0 deletions test/cli.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,15 @@ test('config file changes severities', async () => {
assert.match(r.stdout, /ERROR naming\/vague-id/);
});

test('a config that names a retired rule loads, and says the setting changes nothing', async () => {
const cwd = tempDir();
writeFileSync(join(cwd, 'toolmenu.config.json'), JSON.stringify({ rules: { 'description/late-instruction': 'error' } }));
const r = await run(['snapshot', '--no-write', ...raw('smells')], { cwd });
assert.notEqual(r.code, 2, r.stderr);
assert.match(r.stderr, /rules\["description\/late-instruction"\] changes nothing: retired after 0\.28\.1/);
assert.doesNotMatch(r.stdout, /late-instruction/);
});

test('github format prints workflow annotations', async () => {
const r = await run(['snapshot', '--no-write', '--format', 'github', ...raw('shuffle')]);
assert.match(r.stdout, /^::error title=toolmenu menu\/nondeterministic::/m);
Expand Down
10 changes: 10 additions & 0 deletions test/explain.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,13 @@ test('explain: an unknown rule names the closest; no rule lists them all', async
assert.equal(server.code, 2);
assert.match(server.stderr, /not a server/);
});

test('explain: a retired rule says it was retired and why, in text and JSON', async () => {
const text = await run(['explain', 'description/late-instruction']);
assert.equal(text.code, 0, text.stderr);
assert.match(text.stdout, /^description\/late-instruction: retired after 0\.28\.1: .*no client documents.*descriptionLimit/);
const json = await run(['explain', 'description/late-instruction', '--json']);
assert.equal(json.code, 0, json.stderr);
assert.equal(JSON.parse(json.stdout).id, 'description/late-instruction');
assert.match(JSON.parse(json.stdout).retired, /^retired after 0\.28\.1/);
});
2 changes: 0 additions & 2 deletions test/fixtures/golden/findings.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
{
"snapshot": {
"firecrawl-mcp-3.26.0.json": [
"info description/late-instruction",
"info search/findable firecrawl_scrape",
"info write/no-dry-run",
"warn naming/vague-id"
Expand Down Expand Up @@ -34,7 +33,6 @@
"info write/no-dry-run"
],
"supabase__mcp-server-supabase-0.13.0.json": [
"info description/late-instruction get_publishable_keys",
"info naming/shared-word",
"info write/no-dry-run",
"warn naming/vague-id"
Expand Down
Loading
Loading