fix(tools): strip NetCDF tokens; anchor list_available_output_files to parquet - #23
Merged
Merged
Conversation
…s to parquet Weak Q4 quantized models (observed SimonPu/GLM-4.7-Flash:Q4_K_M) token-match the format name regardless of polarity. With NetCDF named four times across query_files_by_selector's description and `index` arg as negative constraints, plus no parquet anchor on the generically-named list_available_output_files, the model assigned the NetCDF role to list_available_output_files and emitted user-facing prose routing NetCDF prompts there — contradicting the v0.5.0 parquet-only decision. - _tool_descriptions.py: rewrite QUERY_FILES_BY_SELECTOR_DESCRIPTION with "Parquet only." hoisted next to the opener; drop unsupported_format, _excluded_netcdf_count, and mixed-format sentences. Add LIST_AVAILABLE_OUTPUT_FILES_DESCRIPTION ending with "Parquet only." - tools.py:296: route through the new constant. - tools.py:444: drop "NetCDF files do not consume index slots" from the `index` Field description. - test_tool_descriptions.py: drop positive lockstep on unsupported_format and _excluded_netcdf_count (they locked the bug in); add negative `"netcdf" not in desc.lower()` on query_files_by_selector and a new test_list_available_output_files_description_anchor. Runtime envelope keys (`unsupported_format:`, `_excluded_netcdf_count`) are untouched on the wire — the LLM just no longer learns their names from description prose.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_tool_descriptions.py: rewriteQUERY_FILES_BY_SELECTOR_DESCRIPTIONwithParquet only.hoisted next to the opener; dropunsupported_format,_excluded_netcdf_count, and mixed-format sentences. AddLIST_AVAILABLE_OUTPUT_FILES_DESCRIPTIONending withParquet only.tools.py:296routes through the new constant;tools.py:444drops "NetCDF files do not consume index slots" from theindexField description.test_tool_descriptions.py: drop the positive lockstep onunsupported_formatand_excluded_netcdf_count(they locked the bug in); add negative"netcdf" not in desc.lower()onquery_files_by_selectorand a newtest_list_available_output_files_description_anchor.Why
A weak Q4 quantized model (
SimonPu/GLM-4.7-Flash:Q4_K_M) replied to the user's promptlist the last 10 available dates for model cfe_nomwith a hallucinatedquery_datatool and the prose:That's factually wrong —
list_available_output_filesis parquet-only since v0.5.0. The cause is description-token leakage.query_files_by_selectormentioned NetCDF five times across_tool_descriptions.pyandtools.py:444as negative constraints (unsupported_format,_excluded_netcdf_count, "do not consume index slots"). Weak quantized models token-match on the format name regardless of polarity, so "NetCDF unsupported" reads as "supports NetCDF." Combined withlist_available_output_fileshaving no parquet anchor on a generically-named tool, the model assigned the NetCDF role to that tool and surfaced it to the user.Runtime envelope keys (
unsupported_format:,_excluded_netcdf_count) keep emitting on the wire — we just stop teaching the LLM their names from description prose.Generalizes the existing "no concrete example values" convention: no format names of formats we don't support either, even as negative constraints. The rationale is added to the
_tool_descriptions.pymodule docstring.Test plan
.venv-mcp/bin/python -m pytest test_mcp/ -q— 175 passed locallygrep -niE "netcdf|nc4|\.nc[ ,\"']" nextgen_mcp/{prompts,tools,_tool_descriptions}.py— only the module-docstring intentional reference remainslist_available_datesis chosen (model-fitness dependent; this PR does not promise it for sub-7B Q4 models)