v0.5.0: Consolidate query cluster + remove NetCDF support - #20
Merged
Conversation
Generalize 'chatbox-core, Claude Desktop, Cursor, Cline, etc.' to 'MCP clients' — the docstring's load-bearing content is the failure mode, not which clients are affected.
Introduces the v0.5.0 query cluster's single tool replacing the legacy 4-tool surface (query_output_file, query_output_file_from_output_selector, query_output_files_from_output_selector, resolve_output_file). The tool handles three branches via clearly-optional filter args: - file_name set: exact-name lookup in the parquet-filtered list - index set: 0-based into the parquet-filtered list (NetCDF files do not consume index slots) - both None: query all parquet files for the selector Mixed-format selectors silently filter to parquet AND surface the exclusion count as _excluded_netcdf_count on the result envelope when non-zero — preserving R2 'no silent default' parity. Two new envelope classes: - unsupported_format: file_name points at .nc/.nc4 (cheap pre-S3 check) - no_supported_files: selector resolves to zero parquet files Description extracted to _tool_descriptions.py for lockstep contract testing per docs/solutions/best-practices/lockstep-rule-description- string-drift-2026-05-11.md. Lockstep test asserts positive AND negative invariants (no concrete URLs, no inline SQL, no example filenames) per feedback_no_examples_in_tool_descriptions.md. Plan: docs/plans/2026-05-20-001-refactor-nrds-mcps-query-consolidation-plan.md Brainstorm: docs/brainstorms/2026-05-20-nrds-mcps-query-cluster-consolidation-requirements.md
The plot_timeseries slash prompt now drives query_files_by_selector instead of the soon-to-be-deleted query_output_file_from_output_selector. Prompt-surface changes: - Drop the 'index' arg from the user-facing slash signature. The new tool defaults to 'query all parquet files for the selector'; the prompt template instructs the LLM to omit file_name and index so the SQL WHERE feature_id = ... filters across the full union. - 7 args remain (was 8): variable, feature_id, model, forecast, date, cycle, vpu. All still required:true with format hints. - Narrative-only args (variable, feature_id) unchanged. Test fixture lockstep updates: - PLOT_TIMESERIES_ARG_NAMES: drop 'index' - PLOT_TIMESERIES_DESCRIPTIONS: drop 'index' entry - OVERLAPPING_ARG_NAMES: drop 'index' - _selector_tool_schema target string: query_files_by_selector - All assertions referring to the parity tool name updated Plan unit: docs/plans/2026-05-20-001-refactor-nrds-mcps-query-consolidation-plan.md (Unit 3 — Retarget plot_timeseries slash prompt)
The three deleted prompts targeted tools (query_output_file,
resolve_output_file) that are being removed in v0.5.0 — they have no
valid target in the new catalog and serve no purpose under the unified
query_files_by_selector workflow:
- query_by_url: user with an ad-hoc URL can run their own DuckDB query
locally; the server-side direct-URL escape is gone per Compatibility
Policy (NRDS data is fetched via the selector path).
- resolve_file_by_index, resolve_file_by_name: users wanting an S3 URL
read the 'path' field that list_available_output_files already
populates per entry; no separate resolve step needed.
Test fixture changes:
- QUERY_LOOKUP_PROMPTS: shrunk to {lookup_feature} only
- QUERY_LOOKUP_HINTS: dropped s3_url, query, index, file_name entries
- QUERY_LOOKUP_PROMPT_TO_TOOL: dropped 3 mappings
- Historical docstring comment cleanup
29 parametrized prompt tests removed alongside the deleted prompts
(8 prompts/list test + 7 hint-description + 7 bracket-render +
7 substitution-render). lookup_feature parametrization remains.
Plan unit: docs/plans/2026-05-20-001-refactor-nrds-mcps-query-consolidation-plan.md
(Unit 4 — Delete 3 unused slash prompts)
Removes the legacy query cluster now that query_files_by_selector
covers all valid use cases:
Deleted tools (from tools.py + logic.py):
- query_output_file (direct-URL query; no NRDS workflow needs it)
- query_output_file_from_output_selector (singular default-index footgun)
- query_output_files_from_output_selector (replaced by query_files_by_selector)
- resolve_output_file (list_available_output_files already
surfaces the path field per entry)
Deleted dead helpers (from utils_rest.py):
- _duckdb_query_parquet (single-file; only consumer was query_output_file)
- _detect_output_file_kind (parquet-vs-netcdf branch; consumer was
query_output_file)
- _validate_nrds_output_file_url (URL guard for arbitrary external input)
- _normalize_output_file_url (s3:// → https:// translator)
Test migrations:
- test_exception_handling.py: test_query_output_file_returns_envelope_on_binder_exception
→ test_query_files_by_selector_returns_envelope_on_binder_exception
(BinderException recovery path is load-bearing for the new tool too;
_duckdb_query_parquets raises the same exception class)
- test_middleware.py::test_pattern_mismatch_envelope_surfaces_field_description
retargeted from query_output_files_from_output_selector to
query_files_by_selector (same date Field, same pattern)
- test_middleware.py::test_xor_violation_returns_envelope_not_raise
retargeted from resolve_output_file to query_files_by_selector.
Note: the middleware-convert-from-raise path is no longer exercised by
a public tool (query_files_by_selector returns _error_payload directly);
the test now asserts the dict envelope shape rather than the
middleware-flattened string envelope.
- test_query_output_files.py: 4 tests deleted (replaced by test_query_files_by_selector.py)
Documentation:
- nextgen_mcp/README.md tool list updated (9 tools, query section
collapsed to query_files_by_selector + adds get_hydrofabric_pmtiles_layers
which was missing from the original list)
- prompts.py comment block updated to drop reference to deleted
query_by_url / resolve_file_by_* variants and document why
- utils_rest.py docstring example updated from query_output_file's
query arg to query_files_by_selector's query arg
Verified post-merge catalog: 9 tools, 8 prompts (matches plan target).
Plan unit: docs/plans/2026-05-20-001-refactor-nrds-mcps-query-consolidation-plan.md
(Unit 2 — Delete 4 legacy tools + dead helpers + migrate exception test)
Drops NetCDF as a supported format. NRDS NetCDF outputs are still
available in S3; users query them via netCDF-aware tooling (xarray,
h5netcdf, etc.) locally rather than via this server.
Code removed:
- utils_rest._get_troute_df (~15 lines: t-route crosswalk loader)
- utils_rest._duckdb_query_netcdf (~12 lines: in-memory pandas→DuckDB)
- _io_config.open_fsspec_file (~18 lines: NetCDF-only fsspec wrapper;
sole live caller was _get_troute_df)
- utils_rest: 'import xarray as xr' (top-level import)
- logic.py: '_duckdb_query_netcdf' and '_get_troute_df' imports
- logic.py:get_output_file: '.parquet OR .nc' filter trimmed to '.parquet' only
Deps removed:
- xarray (was ~30MB on its own)
- h5netcdf
- (h5py was not in the lock — only optional for h5netcdf)
Docstring/comment updates:
- _io_config.py module docstring: drop 'NetCDF via xarray' from the IO
inventory; module now lists 'S3 via fsspec, DuckDB httpfs' only
- test_query_output_files.py docstring: explain what's left here vs what
moved to test_query_files_by_selector.py and test_exception_handling.py
What remains intentionally:
- The 'NetCDF' references in logic.py:_resolve_parquet_files_for_query
are the unsupported_format: envelope text (telling the LLM what's not
supported and what external tooling to use instead) — NOT live code
references to xarray/h5netcdf modules
- The 'no_supported_files:' envelope still surfaces a netcdf_files count
so users see the scale of the parquet-only filter
- The _excluded_netcdf_count field on result envelopes preserves R2
('no silent default') parity for mixed-format selectors
Plan unit: docs/plans/2026-05-20-001-refactor-nrds-mcps-query-consolidation-plan.md
(Unit 5 — Remove NetCDF code paths + deps)
EXPECTED_MIN_TOOLS: 11 → 9 (post-merge catalog is exactly 9 tools) REMOVED tuple adds the 4 query-cluster tools deleted in v0.5.0: - query_output_file - query_output_file_from_output_selector - query_output_files_from_output_selector - resolve_output_file The existing 4 REMOVED entries (create_plotly_chart_*, build_hydrofabric_feature_map_config, query_hydrofabric_parquet_file) from prior cleanups stay as historical regression guards. REQUIRED tuple swaps query_output_files_from_output_selector (deleted) for query_files_by_selector (added). lookup_hydrofabric_feature remains. REQUIRED_PROMPTS unchanged (plot_timeseries still exists; only its target tool changed). NOTE: query_output_files_from_output_selector moves from REQUIRED to REMOVED in this same commit. Failing to update either side would produce contradictory smoke-gate assertions and block the v0.5.0 deploy on tag push. Both embedded smoke python blocks validated to compile cleanly via local dry-run; YAML parses cleanly. Live verification happens at v0.5.0 tag push via WIF auto-deploy pipeline. Plan unit: docs/plans/2026-05-20-001-refactor-nrds-mcps-query-consolidation-plan.md (Unit 6 — Update CI smoke-gate constants in release.yml)
Documents the breaking changes for external consumers: - 4 tools deleted (query cluster consolidation) - 3 slash prompts deleted (alongside their target tools) - NetCDF support removed (xarray + h5netcdf deps dropped) - 1 new tool (query_files_by_selector) with file_name/index filters - 2 new envelope classes (unsupported_format:, no_supported_files:) - 1 new optional result field (_excluded_netcdf_count) Includes a Migration table mapping each deleted tool to its replacement workflow, plus :v0.4 tag-pinning guidance for :latest consumers per the v0.5.0 Compatibility Policy. Plan unit: docs/plans/2026-05-20-001-refactor-nrds-mcps-query-consolidation-plan.md (Unit 7 — CHANGELOG v0.5.0 entry)
Observed 2026-05-20 against nemotron-3-nano:30b: small models sometimes emit string literals like '<nil>' / 'None' / 'null' when they want to pass None to an Optional[str] arg. Pydantic's str pattern matcher rejected these, the InputValidationEnvelopeMiddleware fired, and the LLM retried with actual None — recovery worked but cost a round-trip. This patch adds a _coerce_none_string BeforeValidator that strips common null-literals to None BEFORE the pattern check. The literal pattern constraint is also dropped (defaulting + validation is handled server-side via 'ensemble or "1"' for medium_range); the LLM still sees ensemble's purpose in the description. Per user clarification 2026-05-20: 'ensemble is only available for medium_range and it is always 1.' Description updated to reflect this (current data uses ensemble=1; defaults to 1 when omitted; ignored for short_range and analysis_assim_extend). The arg is preserved for forward-compat with future ensemble dimensions (historical bucket data included ensemble=16; current data is ensemble=1 only). Applied to both query_files_by_selector_tool and list_available_output_files_tool (same null-literal hazard). Tests: - 12 parametrized unit tests for _coerce_none_string covering all documented null-literals + valid pass-through values - 1 integration test via in-process Client verifies the BeforeValidator is wired correctly on the tool wrapper (ensemble='<nil>' coerced to None at dispatch, tool call succeeds) Per workspace memory feedback_corruption_recovery_via_edit.md, this is a deliberate carve-out from the 'don't add runtime detectors' rule: small-model null-literal emission is a structural-not-semantic syntax leak (the schema rejects, not the content) and the coercion saves a guaranteed round-trip per small-model call without changing behavior for any other model. Plan unit: docs/plans/2026-05-20-001-refactor-nrds-mcps-query-consolidation-plan.md (optional follow-up surfaced during Unit 8 smoke testing against nemotron-3-nano:30b)
Collaborator
Author
Unit 8 pre-deploy smoke validation — local server, 5 prompts across 2 modelsRan the four NRDS canonical workflows against the local server ( Headline outcomes
Validated against v0.5.0 requirements
Recovery patterns exercised
Latest commit on the branch: ensemble null-literal coercion (
|
Code comments now lead with the WHY rather than the dated origin or
plan-number annotation. Detailed history lives in CHANGELOG.md and
docs/.
Removed:
- 'Plan unit:' and 'Plan 2026-MM-XXX' annotations
- 'Observed YYYY-MM-DD against <model>' dated incident attributions
- 'docs/(plans|brainstorms|solutions)/2026-...' path references
- 'feedback_*.md' workspace-memory citations
- 'Bug Nb regression', 'Migrated in v0.5.0' commit-style preambles
- README/docstring backreferences to specific brainstorm/plan files
Kept:
- Technical rationale (WHY each rule exists)
- Test-data values that happen to contain dates (real payload mocks)
- README's 'Claude Desktop' / 'Claude Code CLI' references (real
product names with their actual setup commands)
- Migration markers that name surviving vs deleted tool names
('query_files_by_selector replaces 4-tool query cluster')
Net change ~120 lines removed, no functional change.
Tests: 173/173 pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
query_output_file,query_output_file_from_output_selector,query_output_files_from_output_selector,resolve_output_file) with one unifiedquery_files_by_selectorusingfile_name/indexas clearly-optional filters.query_by_url,resolve_file_by_index,resolve_file_by_name(their target tools are gone).xarray+h5netcdfdeps (~30MB+ Docker image reduction); two new envelope classes (unsupported_format:,no_supported_files:) mark the new format boundary._excluded_netcdf_countwhen non-zero so the parquet-only filter is never silent.This is a breaking release. See
CHANGELOG.mdv0.5.0 entry for the migration table.Commits (squash-on-merge per the plan's PR strategy)
chore: drop product-specific MCP client list from docstring— small unrelated docstring tidy (was already on local main; folded in for branch coherence)feat(query): add unified query_files_by_selector tool— Unit 1feat(prompts): retarget plot_timeseries to query_files_by_selector— Unit 3feat(prompts): delete query_by_url + resolve_file_by_* slash prompts— Unit 4feat(query)!: delete 4 legacy query/resolve tools + dead helpers— Unit 2feat(deps)!: remove NetCDF support and xarray/h5netcdf deps— Unit 5ci: update smoke-gate constants for v0.5.0 catalog— Unit 6docs(changelog): v0.5.0 entry— Unit 7Verification
file_name, negative/out-of-rangeindex,unsupported_format:/no_supported_files:envelopes, mixed-format_excluded_netcdf_count, parquet-only index semantics).test_tool_descriptions.pyenforces positive AND negative invariants onquery_files_by_selector's description (no concretes3://URLs, no example filenames, no inline SQL — perfeedback_no_examples_in_tool_descriptions.md).Client(mcp)confirms 9 tools, 8 prompts, all 8 deleted tool names absent.release.ymlPython script against the in-process Client passes (assertsEXPECTED_MIN_TOOLS=9, REMOVED-set absent, REQUIRED-set present, REQUIRED_PROMPTS present).Test plan
pytestfull suite (160 pass)release.ymlconstants)Migration
External consumers hardcoded to deleted tool names break at deploy — accepted per the v0.5.0 Compatibility Policy (
docs/plans/2026-05-20-001-...-plan.mdand CHANGELOG). External consumers using:latestDocker tag will hit this at the next pull; pin to:v0.4to defer (tag availability to be confirmed in registry).Links
docs/plans/2026-05-20-001-refactor-nrds-mcps-query-consolidation-plan.mddocs/brainstorms/2026-05-20-nrds-mcps-query-cluster-consolidation-requirements.md/ce:debug2026-05-20 (conversation history)🤖 Generated with Claude Code