Skip to content

Validate tool calls and return detailed errors to improve tool call success rate - #783

Open
choongng wants to merge 2 commits into
huggingface:mainfrom
choongng:upstream/tool-argument-schema
Open

choongng wants to merge 2 commits into
huggingface:mainfrom
choongng:upstream/tool-argument-schema

Conversation

@choongng

@choongng choongng commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

This PR checks a tool call's arguments against the tool's declared schema, before the tool runs, and answers a call that doesn't fit with a single error that lists each problem with enough specificity to allow recovery in few attempts, for example:

Invalid arguments for read: missing required "path"; "limit" must be integer, got string; unknown argument "colour". Send the call again with these fixed.

The checker covers the keywords tau's own tools use (type, required, properties, additionalProperties, enum, items) as well other tools in my workflow, and the built-in tool definitions have been adjusted to match their actual existing behavior. Tests document the precise behavior.

Implementation: This adds about 130 lines in tau_agent, rather than using an existing library like jsonschema, as this narrower implementation is a good fit for the type of human/robot comprehensible errors we want to construct.

The second commit makes bash's description optional, some models (including Gemma 4) tend to leave it out which with validation would result in excessive validation refusals.

All tests pass for each commit. The impact this has varies by model and model serving platform, with server-side schema validation this should never trigger but with local models and something like llama-server it makes a big difference. Details are in dev-notes/tool-argument-validation.md.

Choong Ng added 2 commits October 28, 2026 02:46
`description` only labels the call in the UI, and the bash tool already runs
without it; the TUI then shows "Running shell command" instead. Listing it as
required tells models they must send it, and some (Gemma 4 among them) leave
it out anyway. The tools reference now lists it as optional.
Before a tool runs, the agent loop checks the call's arguments against the
tool's `parameters` schema (`type`, `required`, `properties`,
`additionalProperties`, `enum`, `items`) and answers a call that does not fit
with one error listing every problem and its path; the tool does not run.
A model that gets every problem at once can fix the call in one retry, rather
than learning one problem per refused call.

Valid arguments pass unchanged, other keywords and an empty schema are not
checked, an optional argument sent as null counts as not sent, and unparsed
`_raw_arguments` are left to the tool.

The loop now runs a tool's `prepare_arguments` before the check and passes its
result to the tool. The edit tool uses it to turn a top-level
`oldText`/`newText` pair, or `edits` sent as a JSON string, into the `edits`
list its schema declares.
@choongng choongng changed the title Validate tool calls and provide detailed parse errors to improve tool call success rate Validate tool calls and return detailed errors to improve tool call success rate Oct 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant