Repository navigation
Echo model-derived diagnostics through the coarse error taxonomy #24
Description
Problem
Every failure a model sees through the MCP boundary is one of eleven bare sentinel strings (docs/docs/reference/mcp-tools.md, Errors table). Three cases make the agent authoring loop needlessly blind:
- Syntax error →
invalid program. No line, no column. The model must bisect its own program by resubmission. - Binding failure →
invalid capability arguments. No argument name. The model cannot tell a typo'd keyword from a type mismatch from a missing required field. printoutput is (correctly) discarded, so with no error detail there is no debugging channel at all — failed programs are blind retries that burn turns andexecutebudget.
The detail already exists at every layer and is discarded by the projection chain (internal/execution/execute.go: classifyRuntimeError → server.go: projectExecutionError → mcpserver/server.go: projectToolError):
- Parse errors arrive as
syntax.Errorwith 1-basedLine/Colpositions; unresolved names asresolve.ErrorList, also position-stamped (verified against the pinnedgo.starlark.netrevision). - Runtime errors arrive as
*starlark.EvalErrorcarrying aCallStackwith frame positions in the model's own source. internal/binding/input.goalready writes exactly the messages we want the model to see:unknown argument "keu",argument "limit" must be an integer or None,missing required argument "key",positional arguments are not accepted.
Principle
Echo only information the model already possesses. Positions in its own source and the names/types of its own arguments disclose nothing about the host. Host-derived text — handler error messages, policy diagnostics, panic values, Go stack detail — stays hidden exactly as today. The non-disclosure posture for host internals is correct and must not be weakened.
Proposed selectivity
| Error class | Echo to the model |
|---|---|
| Syntax / resolve error | source position + parser/resolver message (entirely model-derived), e.g. invalid program: <codemode>:3:7: got '=', want primary expression |
| Binding failure | the existing internal/binding message including the argument name, e.g. invalid capability arguments: unknown argument "keu" |
| Handler failure, policy failure, internal failure | coarse sentinel exactly as today; optionally attach only the call-site position from the CallStack ("line 12 of your program"), never the wrapped cause |
| Resource limits, auth, not-found | unchanged |
Danger note (why the blanket flattening exists)
starlark.EvalError.Msg embeds the wrapped cause. For handler failures that cause is host error text — exactly what must stay hidden. So pass-through must be type-selective, not blanket. The branch points already exist: classifyRuntimeError unwraps with errors.Is against each sentinel class today; the change is carrying a safe detail string alongside the sentinel for the model-derived classes instead of dropping it.
Compatibility
Keep the stable coarse sentinel text as a prefix (invalid program: ..., invalid capability arguments: ...) so clients and tests keyed on categories keep matching. The errors table and the "Authoring and recovery" section in docs/docs/reference/mcp-tools.md must be updated in the same PR, since the listed descriptions are the model-facing contract.
Acceptance sketch
- Submitting source with a syntax error returns
invalid programwith the position of the error in the submitted source. - Calling a capability with an unknown keyword returns
invalid capability argumentswith that keyword's name. - A handler that returns
errors.New("db password rejected")still yields exactlycapability failed— an e2e test must assert the host text does not appear in any content item. - Policy diagnostics, panic values, and stack details remain absent from all error payloads (existing non-disclosure tests keep passing).
Origin
Product/UX review of the current surface at 8b5302b. Related but independent: the binding-matrix issue (filed separately) — both target the model-side authoring loop.
Activity
jmgilman commentedon Aug 25, 2026
Fresh evidence from a live end-to-end UX review at
a5031b5(real stdio server, real MCP client, worker execution):Writing the first plausible composite program, I used
sum(...)— a Python builtin that Starlark does not have. The resolver produces a position-stamped "undefined: sum" internally; the client sees onlyinvalid program. Diagnosing it required knowledge of the starlark-go universe, which a model in a recovery loop does not have. The "Authoring and recovery" checklist inmcp-tools.mddoes not cover the undefined-name failure class at all, so the documented recovery steps cannot resolve the most Python-habituated failure mode.Two updates for whoever implements this:
The error boundary is now a documented contract, not just code. PR docs(security): define exact error boundary #22 formalized in
mcp-tools.md("does not expose … argument names or values, source locations or text") and the security model that source locations are never exposed. Implementing this issue therefore requires deliberately amending that documented contract for the model-derived classes (parse/resolve positions, binding argument names), not silently changing behavior. The host-derived classes (handler messages, policy detail, panic values) stay non-disclosing as written.Cheap complementary fix with zero contract tension: the
executetool description and reference nowhere state which language surface exists. A short "Starlark, not Python: nosum, noimport/while/f-strings; available builtins are …" note in the model-facing contract would proactively prevent the most common failure class even before diagnostics land.