From 7aad443de3262e97d66334d1cd31d8ddaca03f80 Mon Sep 17 00:00:00 2001 From: Harsh Singh Date: Mon, 10 Aug 2026 01:55:23 +0530 Subject: [PATCH] feat(mcp): budget the tools MCP servers put in front of the model MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MCP was the largest standing-context line item kitbash did not measure — the one place where "we measure token cost" had an undefended hole. A skill body is statically countable, which is why context.budget can be enforced. An MCP server's real cost is not: it is the JSON schema of every tool the server exposes, and knowing that means asking the server — which for stdio means executing it. That is what the install gate exists to prevent, so kitbash does not connect, and does not print an estimate it cannot derive. What it counts exactly is what the manifest already states. Every tool in a declared allowlist is a tool definition carried for the whole session, so the allowlist is an exact floor. compile and doctor now report it, and say plainly that the token cost is unmeasured and why. A server declaring tools = ["*"] has no countable bound, so the total renders as "8+ (1 server(s) declare "*")" with a warning that it cannot be bounded, rather than folding into a total that would be wrong. max_mcp_tools in [policy] caps the declared total; a breach warns and fails --strict. The default is 100 — the only primary-source numeric limit across the clients (Windsurf documents it as a hard cap past which tools are dropped). The widely repeated Cursor "40 tool" limit is deliberately not encoded: it appears only in forum posts and blogs, never in Cursor's docs, and a tool whose pitch is measurement must not ship a folklore number. Adds 9 tests. 0.20.0. --- CHANGELOG.md | 12 +++++++ README.md | 7 ++++ packages/cli/package.json | 2 +- packages/cli/scripts/test.mjs | 47 ++++++++++++++++++++++++++ packages/cli/src/commands.ts | 14 +++++++- packages/cli/src/mcp.ts | 63 +++++++++++++++++++++++++++++++++++ site/changelog.html | 16 +++++++-- site/index.html | 2 +- 8 files changed, 158 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 93ab9e3..8dfe509 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,18 @@ Format: [Keep a Changelog](https://keepachangelog.com). Versioning: semver — for skills *and* for this CLI, breaking prompt changes are breaking changes. +## [0.20.0] — 2026-08-09 + +The token-cost argument, applied to MCP — the largest standing-context line item Kitbash was not measuring. + +A skill body's cost is statically countable, which is why `context.budget` can be enforced at compile time. An MCP server's real cost is not: the tokens it adds are the JSON schema of every tool it exposes, and knowing those means asking the server — which for a stdio server means **executing it**. That is exactly what the install gate exists to prevent, so Kitbash does not do it, and does not print an estimate it cannot derive. + +What it can count exactly is the thing the manifest already states. Every tool in a declared allowlist is a tool definition the agent carries for the whole session, so the allowlist is an exact **floor** on the cost. + +### Added +- **MCP tool budget, reported on every compile and by `doctor`** — `MCP tool budget: 8 tool(s) across 2 server(s); cap 100`, alongside an explicit statement that the token cost is unmeasured and why. A server declaring `tools = ["*"]` has no countable bound, so the total renders as `8+ (1 server(s) declare "*")` and warns that it cannot be bounded, rather than folding into a total that would be wrong. +- **`max_mcp_tools` in `[policy]`** — caps declared MCP tools across all skills. A breach warns and fails `--strict`. The default ceiling is **100**, the one primary-source numeric limit found across the clients (Windsurf documents it as a hard cap past which tools are dropped). A widely-repeated "40 tool" Cursor limit is deliberately **not** encoded: it appears only in forum posts and blogs, never in Cursor's own docs, and a tool whose pitch is measurement must not ship a folklore number. + ## [0.19.1] — 2026-08-09 Follow-through on 0.19.0: policy gets a say over MCP servers, and the support matrix becomes something you can read before you compile. diff --git a/README.md b/README.md index fae5391..9ec58f9 100644 --- a/README.md +++ b/README.md @@ -168,6 +168,7 @@ deny_write = true # refuse skills declaring write access max_budget = 6000 # cap per-skill context budget # deny_mcp = true # refuse skills declaring any MCP server # allow_mcp_servers = ["https://mcp.your-org.com/*"] # globs; matched against a server's command or url +# max_mcp_tools = 100 # cap declared MCP tools across all skills # deny_remote_exec = false # opt out of the curl|sh body lint (default: on) ``` @@ -190,6 +191,12 @@ Three targets have a project-scoped MCP config file, and those are the three Kit The other eight targets get a warning naming the specific reason — `no-mcp-surface` (aider, AGENTS.md have no configuration mechanism), `no-project-scope` (Cline and Windsurf are user-global only), `needs-shared-file-merge` (Zed and Gemini keep servers in a settings file full of unrelated user config) — and **no file**. A config the client never reads would look configured and do nothing, which is the failure mode this project exists to prevent. +### What it costs you + +`compile` and `doctor` report the tool budget: `MCP tool budget: 8 tool(s) across 2 server(s); cap 100`. Every tool in a declared allowlist is a tool definition the agent carries for the whole session, so the allowlist is an exact floor on the cost — and `max_mcp_tools` in `[policy]` caps it, warning on breach and failing `--strict`. The default ceiling is 100, the one limit any client documents (Windsurf, past which tools are dropped). + +The *real* token cost is reported as unmeasured, and stays that way on purpose. It is the JSON schema of every tool a server exposes, which is only knowable by asking the server — and asking a stdio server means executing it, which is precisely what the install gate exists to prevent. A floor Kitbash can prove beats an estimate it cannot. + A declared MCP server is third-party code that will run with your agent's permissions, so its lints are non-bypassable, like the four safety lints: shell strings, unpinned versions, literal credentials, plain-`http` URLs off loopback, and a missing `tools` allowlist all block install, `--yes` included. Secrets may only appear as `${VAR}`; where a format defines no credential reference and expands no variables, the server is omitted with a warning rather than emitted broken. ## In CI diff --git a/packages/cli/package.json b/packages/cli/package.json index b2dddfa..0ebe390 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "kitbash", - "version": "0.19.1", + "version": "0.20.0", "description": "The package manager and compiler for AI agent skills — write once, run in every coding agent", "license": "Apache-2.0", "author": "Harsh Singh", diff --git a/packages/cli/scripts/test.mjs b/packages/cli/scripts/test.mjs index 46e0e2d..891f277 100644 --- a/packages/cli/scripts/test.mjs +++ b/packages/cli/scripts/test.mjs @@ -1632,6 +1632,53 @@ try { rmSync(polTmp, { recursive: true, force: true }); } +// ── MCP tool budget ────────────────────────────────────────────────────────── +// The standing-cost argument applied to MCP. Counted from the declared allowlist +// (an exact floor), never estimated — measuring the real cost means executing the +// server, which is what the install gate exists to prevent. +const budTmp = mkdtempSync(join(tmpdir(), "kitbash-budget-")); +try { + const bs = join(budTmp, "src"); + mkdirSync(bs, { recursive: true }); + writeFileSync( + join(bs, "skill.toml"), + '[skill]\nname = "toolheavy"\nversion = "1.0.0"\ndescription = "Declares several MCP tools across two servers"\n[context]\nbudget = 1500\n\n[mcp.servers.a]\ntransport = "stdio"\ncommand = "npx"\nargs = ["-y", "@acme/a@1.0.0"]\ntools = ["t1","t2","t3","t4","t5"]\n\n[mcp.servers.b]\ntransport = "streamable-http"\nurl = "https://b.example.com/mcp"\ntools = ["u1","u2","u3"]\n', + ); + writeFileSync(join(bs, "SKILL.md"), "# Tool heavy\n\nBody.\n"); + writeFileSync(join(budTmp, "kitbash.toml"), '[project]\ntargets = ["claude-code"]\n'); + run(["install", `file:${bs}`, "--yes"], budTmp); + + const def = run(["compile"], budTmp); + check("budget: the tool count is reported on every compile", def.out.includes("MCP tool budget: 8 tool(s) across 2 server(s)"), def.out); + check("budget: token cost is reported as unmeasured, not guessed", def.out.includes("unmeasured"), def.out); + check("budget: under the cap is a note, not a warning", def.status === 0 && !def.out.includes("budget exceeded"), def.out); + + writeFileSync(join(budTmp, "kitbash.toml"), '[project]\ntargets = ["claude-code"]\n[policy]\nmax_mcp_tools = 5\n'); + const over = run(["compile"], budTmp); + check("budget: a breach of max_mcp_tools warns", over.out.includes("MCP tool budget exceeded"), over.out); + check("budget: a breach still compiles by default", over.status === 0, over.out); + const strictBudget = run(["compile", "--strict"], budTmp); + check("budget: a breach fails --strict", strictBudget.status === 1, strictBudget.out); + + // A wildcard server cannot be bounded — say so rather than fold it into a total. + const ws = join(budTmp, "wild"); + mkdirSync(ws, { recursive: true }); + writeFileSync( + join(ws, "skill.toml"), + '[skill]\nname = "wildy"\nversion = "1.0.0"\ndescription = "Declares a wildcard MCP tool allowlist"\n[context]\nbudget = 1500\n\n[mcp.servers.wild]\ntransport = "streamable-http"\nurl = "https://w.example.com/mcp"\ntools = ["*"]\n', + ); + writeFileSync(join(ws, "SKILL.md"), "# Wild\n\nBody.\n"); + writeFileSync(join(budTmp, "kitbash.toml"), '[project]\ntargets = ["claude-code"]\n'); + run(["install", `file:${ws}`, "--yes"], budTmp); + const wild = run(["compile"], budTmp); + check("budget: a wildcard server makes the total a floor, marked +", wild.out.includes('8+ (1 server(s) declare "*")'), wild.out); + check("budget: and says it cannot be bounded", wild.out.includes("cannot be bounded"), wild.out); + const wdoc = run(["doctor"], budTmp); + check("budget: doctor reports the same budget line", wdoc.out.includes("MCP tool budget:"), wdoc.out); +} finally { + rmSync(budTmp, { recursive: true, force: true }); +} + if (failures) { console.error(`\n${failures} test(s) failed`); process.exit(1); diff --git a/packages/cli/src/commands.ts b/packages/cli/src/commands.ts index f7af5cb..256e9f9 100644 --- a/packages/cli/src/commands.ts +++ b/packages/cli/src/commands.ts @@ -11,7 +11,7 @@ import { dropLock, integrityOf, readLock, upsertLock, walk, LOCK_FILE } from "./ import { fileChanges, manifestDelta, textOf, unifiedDiff } from "./diff.js"; import { collectImports, driftGroups, type ImportedSource } from "./importers.js"; import { estimateTokens, loadInstalledSkills, loadInstalledSkillsSafe, loadSkill, resolveBody, schemaLints, standingStub, COMMAND_RE, NAME_RE, SKILLS_DIR, type LoadedSkill } from "./ksf.js"; -import { collectServers, emitMcp, mcpLints, mcpSupportMatrix, mcpWarnings } from "./mcp.js"; +import { collectServers, DEFAULT_MAX_TOOLS, emitMcp, mcpLints, mcpSupportMatrix, mcpWarnings, toolBudget, toolBudgetReport } from "./mcp.js"; import { toSarif, type SarifFinding } from "./sarif.js"; import { parseToml } from "./toml.js"; @@ -42,6 +42,7 @@ const INIT_CONFIG = `# kitbash project configuration — https://github.com/sing # max_budget = 6000 # refuse skills with a larger context budget # deny_remote_exec = false # opt OUT of the download-and-execute body lint (default: on) # deny_mcp = true # refuse skills declaring any MCP server +# max_mcp_tools = 100 # cap declared MCP tools across all skills (default: 100) # allow_mcp_servers = ["https://mcp.your-org.com/*"] # globs; matched against a server's command or url `; @@ -641,6 +642,7 @@ export async function cmdDoctor(): Promise { const declared = collectServers(skills).servers; if (declared.length) { console.log(`\nMCP servers declared: ${declared.map((s) => s.name).join(", ")}`); + console.log(` ${toolBudgetReport(toolBudget(declared), loadPolicy(root)?.maxMcpTools ?? DEFAULT_MAX_TOOLS).line}`); for (const line of mcpSupportMatrix()) console.log(` ${line}`); } @@ -721,6 +723,8 @@ interface Policy { denyRemoteExec: boolean; /** Refuse any skill that declares an MCP server at all. */ denyMcp: boolean; + /** Cap on declared MCP tools across all skills. Absent uses the documented 100-tool ceiling. */ + maxMcpTools?: number | undefined; /** * Globs an MCP server's command (stdio) or url (remote) must match. Empty means * unrestricted — the same shape as allow_sources, so an org can pin servers to @@ -747,6 +751,7 @@ function loadPolicy(root: string): Policy | null { // Absent means true — you must opt OUT of the remote-exec block explicitly. denyRemoteExec: tbl["deny_remote_exec"] !== false, denyMcp: tbl["deny_mcp"] === true, + maxMcpTools: typeof tbl["max_mcp_tools"] === "number" ? (tbl["max_mcp_tools"] as number) : undefined, allowMcpServers: Array.isArray(tbl["allow_mcp_servers"]) ? (tbl["allow_mcp_servers"] as unknown[]).filter((x): x is string => typeof x === "string") : [], @@ -957,6 +962,13 @@ export async function cmdCompile(args: string[]): Promise { return 1; } if (mcpServers.length) { + // The standing-cost argument, applied to MCP. Counted from the declared + // allowlist — an exact floor, never an estimate — and reported as a note so + // the number shows on every compile, with a breach as a real warning. + const report = toolBudgetReport(toolBudget(mcpServers), loadPolicy(root)?.maxMcpTools ?? DEFAULT_MAX_TOOLS); + notes.push(report.line); + warnings.push(...report.warnings); + const unsupported: string[] = []; for (const adapter of adapters) { const out = emitMcp(adapter.id, mcpServers, AGENT_PLUGIN_DIR); diff --git a/packages/cli/src/mcp.ts b/packages/cli/src/mcp.ts index 8853f56..d5703f7 100644 --- a/packages/cli/src/mcp.ts +++ b/packages/cli/src/mcp.ts @@ -394,6 +394,69 @@ function emitCopilot(servers: McpServer[]): McpEmit { return { files: [{ path: ".github/mcp.json", content: json({ mcpServers: out }) }], warnings }; } +// ── tool budget ────────────────────────────────────────────────────────────── + +/** + * The tool budget: how many MCP tools a repo's skills put in front of the model. + * + * This is the standing-cost argument applied to MCP. A skill body's cost is + * statically countable, which is why Kitbash can enforce `context.budget`. An + * MCP server's real cost — the JSON schema of every tool it exposes — is not + * knowable without asking the server, and asking a stdio server means executing + * it. That is precisely what the install gate exists to prevent, so Kitbash does + * not do it, and does not estimate a number it cannot derive. + * + * What it does count is the thing the manifest already states: the declared tool + * allowlist. Every one of those tools is a tool definition the agent carries, so + * the count is a real floor on the cost, and it is exact whenever the allowlist + * is exact. A server declaring `["*"]` has no countable bound — that is reported + * as unbounded rather than folded into a total that would then be wrong. + */ +export interface ToolBudget { + /** Tools named explicitly across all servers — an exact floor on what is loaded. */ + declared: number; + /** Servers declaring `["*"]`: real cost unknown without asking the server. */ + unbounded: string[]; + servers: number; +} + +/** + * Windsurf documents a hard cap of 100 tools total; past it, tools are dropped. + * It is the only primary-source numeric limit found across the clients, so it is + * the default ceiling. (A widely-repeated Cursor "40 tool" limit appears only in + * forum posts and blogs, never in Cursor's own docs, so it is deliberately not + * encoded here — a tool whose pitch is measurement must not ship a folklore number.) + */ +export const DEFAULT_MAX_TOOLS = 100; + +export function toolBudget(servers: McpServer[]): ToolBudget { + let declared = 0; + const unbounded: string[] = []; + for (const s of servers) { + if (s.tools.includes("*")) unbounded.push(s.name); + else declared += s.tools.length; + } + return { declared, unbounded, servers: servers.length }; +} + +/** + * Report the budget, and fail when a countable total breaches the cap. Servers + * declaring `["*"]` can never be proven under a cap, so they are surfaced as the + * reason the total is a floor rather than silently passing. + */ +export function toolBudgetReport(b: ToolBudget, max: number): { line: string; warnings: string[]; over: boolean } { + const bound = b.unbounded.length ? `${b.declared}+ (${b.unbounded.length} server(s) declare "*")` : `${b.declared}`; + const line = `MCP tool budget: ${bound} tool(s) across ${b.servers} server(s); cap ${max}. Real token cost is unmeasured — it needs the server's tool schemas, and reading those means running it.`; + const warnings: string[] = []; + const over = b.declared > max; + if (over) { + warnings.push(`MCP tool budget exceeded: ${b.declared} declared tools against a cap of ${max}. Windsurf documents 100 total tools as a hard cap, past which tools are dropped — and every tool definition is carried in context for the whole session.`); + } else if (b.unbounded.length) { + warnings.push(`MCP tool budget cannot be bounded: ${b.unbounded.join(", ")} declare tools = ["*"], so the total is at least ${b.declared} and may exceed the cap of ${max}. Name the tools to make it countable.`); + } + return { line, warnings, over }; +} + /** Targets that can carry an MCP declaration today. */ const MCP_TARGETS = ["agent-plugins", "claude-code", "copilot"]; diff --git a/site/changelog.html b/site/changelog.html index 8264471..dab8f39 100644 --- a/site/changelog.html +++ b/site/changelog.html @@ -91,7 +91,7 @@

Changelog

Releases follow Keep a Changelog and semver — for skills and for this CLI, breaking prompt changes are breaking changes. The CLI is published to npm as kitbash and to Homebrew via singhharsh1708/tap. Tagged builds are on the GitHub releases page.

-
v0.19.1Current CLI version
+
v0.20.0Current CLI version
8Compile targets
Apache-2.0License
@@ -105,10 +105,22 @@

Changelog

Confirm with kitbash --version, which reads the installed package.json. Install and uninstall routes are covered on the installation page.

+
+
+

v0.20.0

+ 2026-08-09latest +
+

The token-cost argument, applied to MCP — the largest standing-context line item Kitbash was not measuring.

+

A skill body's cost is statically countable, which is why context.budget can be enforced at compile time. An MCP server's real cost is not: the tokens it adds are the JSON schema of every tool it exposes, and knowing those means asking the server — which for a stdio server means executing it. That is exactly what the install gate exists to prevent, so Kitbash does not do it, and does not print an estimate it cannot derive.

+

What it can count exactly is the thing the manifest already states. Every tool in a declared allowlist is a tool definition the agent carries for the whole session, so the allowlist is an exact floor on the cost.

+

Added

+
  • MCP tool budget, reported on every compile and by doctor — MCP tool budget: 8 tool(s) across 2 server(s); cap 100, alongside an explicit statement that the token cost is unmeasured and why. A server declaring tools = ["*"] has no countable bound, so the total renders as 8+ (1 server(s) declare "*") and warns that it cannot be bounded, rather than folding into a total that would be wrong.
  • max_mcp_tools in [policy] — caps declared MCP tools across all skills. A breach warns and fails --strict. The default ceiling is 100, the one primary-source numeric limit found across the clients (Windsurf documents it as a hard cap past which tools are dropped). A widely-repeated "40 tool" Cursor limit is deliberately not encoded: it appears only in forum posts and blogs, never in Cursor's own docs, and a tool whose pitch is measurement must not ship a folklore number.
+
+

v0.19.1

- 2026-08-09latest + 2026-08-09

Follow-through on 0.19.0: policy gets a say over MCP servers, and the support matrix becomes something you can read before you compile.

Added

diff --git a/site/index.html b/site/index.html index 210bd4a..3506688 100644 --- a/site/index.html +++ b/site/index.html @@ -151,7 +151,7 @@ -

Open format for AI agent skills · v0.19.1 · stable spec (RFC 0002)

+

Open format for AI agent skills · v0.20.0 · stable spec (RFC 0002)

Write an agent skill once. Run it everywhere.

Get started