fix(tools): bias create_*/data toward structured form + fix_hint on JSON-decode failure - #6
Open
romer8 wants to merge 1 commit into
Open
fix(tools): bias create_*/data toward structured form + fix_hint on JSON-decode failure#6romer8 wants to merge 1 commit into
romer8 wants to merge 1 commit into
Conversation
…SON-decode failure Root cause of the 2026-05-18 200s incident: the LLM emitted `data` as a JSON-string-literal containing malformed JSON (240 rows × ~70 chars, token-by-token generation drifted mid-array on a stray comma). The tool body's json.loads() rejected with `invalid_args: data is not valid JSON: ...` — but the rejection envelope carried no recovery guidance, so the LLM regenerated the entire stringified array (~63s of wasted output tokens) and frequently hit the same drift on retry. Two changes, both within the existing Union[List, str] schema (no breaking change for models that always emit JSON strings): 1) Tool descriptions on create_plotly_chart and create_data_table now STRONGLY PREFER the structured-array primary type and warn that the stringified form is error-prone for long arrays + silently doubles latency on parse failure. Native tool-calling on Ollama/OpenAI/ Anthropic grammars constrain the structured form's outer shape; they cannot constrain the content of a string value, so biasing the LLM toward the structured form is the actual fix for the token-drift class. 2) The JSONDecodeError envelope from the tool body now carries a `fix_hint` field directing the LLM to retry with the structured form. Mirrors the input-validation middleware's recovery pattern so the LLM gets actionable guidance on first failure. Companion to chatbox-core's planned JSON-mode work (plan docs/plans/2026-05-18-003-feat-mcp-tool-call-json-mode-plan.md, in the firoh workspace) — that plan adds OpenAI strict:true on the client side. This server-side change makes the structured form attractive across ALL providers without requiring provider-specific client support. Tests: 2 new regression tests in test_tool_input_robustness.py pinning the fix_hint contract on both tools. Suite 786 → 788 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Companion server-side fix to the 2026-05-18 200s production incident.
The LLM emitted `data` as a JSON-string-literal containing malformed JSON (240-row token-by-token generation drifted mid-array). The tool body rejected with `invalid_args: data is not valid JSON` — but the envelope carried no recovery guidance, so the LLM regenerated the entire stringified array (~63s wasted) and frequently hit the same drift on retry.
Two changes, both within the existing `Union[List, str]` schema (no breaking change for models that always emit JSON strings):
Why this works across all providers
Biasing the LLM toward the structured form is the actual fix; the previous JSON-mode plan's idea of passing schemas as `format` parameters doesn't work because schema-constrained generation cannot validate the content of a string value.
Tests
Companion work
Test plan