Skip to content

fix(tools): bias create_*/data toward structured form + fix_hint on JSON-decode failure - #6

Open
romer8 wants to merge 1 commit into
mainfrom
feat/data-arg-structured-bias
Open

fix(tools): bias create_*/data toward structured form + fix_hint on JSON-decode failure#6
romer8 wants to merge 1 commit into
mainfrom
feat/data-arg-structured-bias

Conversation

@romer8

@romer8 romer8 commented May 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Companion server-side fix to the 2026-05-18 200s production incident.

The LLM emitted `data` as a JSON-string-literal containing malformed JSON (240-row token-by-token generation drifted mid-array). The tool body rejected with `invalid_args: data is not valid JSON` — but the envelope carried no recovery guidance, so the LLM regenerated the entire stringified array (~63s wasted) and frequently hit the same drift on retry.

Two changes, both within the existing `Union[List, str]` schema (no breaking change for models that always emit JSON strings):

  1. Description bias. `create_plotly_chart` and `create_data_table` descriptions now STRONGLY PREFER the structured-array primary type and warn that the stringified form is error-prone for long arrays + silently doubles latency on parse failure.
  2. `fix_hint` on JSONDecodeError. The rejection envelope now carries a `fix_hint` directing the LLM to retry with the structured form. Mirrors the existing input-validation middleware recovery pattern.

Why this works across all providers

  • Ollama native tool-calling constrains tool_call argument STRUCTURE via the model's grammar. If the LLM picks the structured arm (which the description now strongly prefers), grammar-constrained generation prevents mid-array drift entirely.
  • OpenAI strict:true (separate chatbox-core PR) enforces the same on the client side.
  • Anthropic validates tool calls server-side against the schema.

Biasing the LLM toward the structured form is the actual fix; the previous JSON-mode plan's idea of passing schemas as `format` parameters doesn't work because schema-constrained generation cannot validate the content of a string value.

Tests

  • 2 new regression tests in `test_tool_input_robustness.py` pinning the `fix_hint` contract on both tools.
  • Suite: 786 → 788 passed.

Companion work

  • chatbox-core PR (pending): OpenAI strict:true on tool definitions in the OpenAI adapter. Telemetry hook for measurement.
  • Origin plan: `docs/plans/2026-05-18-003-feat-mcp-tool-call-json-mode-plan.md` in the firoh workspace (re-scoped 2026-05-18 after corrected understanding of Ollama `format` semantics).

Test plan

  • `pytest test_mcp/` → 788 passed
  • Live re-run of the 200s scenario (240-row time-series → create_plotly_chart) — confirm the LLM picks the structured form on first attempt OR recovers via the new `fix_hint` on second attempt

…SON-decode failure

Root cause of the 2026-05-18 200s incident: the LLM emitted `data` as a
JSON-string-literal containing malformed JSON (240 rows × ~70 chars,
token-by-token generation drifted mid-array on a stray comma). The
tool body's json.loads() rejected with `invalid_args: data is not
valid JSON: ...` — but the rejection envelope carried no recovery
guidance, so the LLM regenerated the entire stringified array (~63s
of wasted output tokens) and frequently hit the same drift on retry.

Two changes, both within the existing Union[List, str] schema (no
breaking change for models that always emit JSON strings):

1) Tool descriptions on create_plotly_chart and create_data_table now
   STRONGLY PREFER the structured-array primary type and warn that the
   stringified form is error-prone for long arrays + silently doubles
   latency on parse failure. Native tool-calling on Ollama/OpenAI/
   Anthropic grammars constrain the structured form's outer shape;
   they cannot constrain the content of a string value, so biasing
   the LLM toward the structured form is the actual fix for the
   token-drift class.

2) The JSONDecodeError envelope from the tool body now carries a
   `fix_hint` field directing the LLM to retry with the structured
   form. Mirrors the input-validation middleware's recovery pattern
   so the LLM gets actionable guidance on first failure.

Companion to chatbox-core's planned JSON-mode work (plan
docs/plans/2026-05-18-003-feat-mcp-tool-call-json-mode-plan.md, in
the firoh workspace) — that plan adds OpenAI strict:true on the
client side. This server-side change makes the structured form
attractive across ALL providers without requiring provider-specific
client support.

Tests: 2 new regression tests in test_tool_input_robustness.py pinning
the fix_hint contract on both tools. Suite 786 → 788 passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant