Read a wide schema the way real databases actually demand - #33
Merged
Conversation
A wide schema spent its column-hint probe budget on whichever tables were introspected first, so every table past roughly the fiftieth got no hints at all. The budget is now a fair share per table instead. The pruner's table cap was a fixed 40 regardless of the token budget, dropping a table the budget had room for. The cap now only guards a pathological schema; the budget decides what is sent, and one wide table can no longer evict every smaller table behind it in the same pass. A result whose row limit was lowered - rather than added where there was none - was not counted as truncated, so a "top 5000" that came back capped at 100 rows reported no warning. Both cases are now caught, on the flag and the message. DuckDB stopped silently truncating a schema past 100,000 total columns, and now rejects an oversized .sql upload before reading it into memory rather than risking an out-of-memory crash partway through. Oracle's JSON-array hint stopped emitting an unquoted identifier the database would reject. MongoDB now warns when a collection's field count passes the describe cap, instead of silently naming only the first 500.
sqlite-jdbc answers a whole-schema getColumns() with one UNION ALL term per column, and SQLite refuses a compound SELECT past 500 terms. An ordinary Android app database - a couple hundred tables, a handful of columns each - crosses that without trying, and failed to import at all with a message that read as a corrupt file. Columns are now read with PRAGMA table_info, one table at a time, which is what the npm package has always done and is why it was never affected. The same JDBC round-trip cost applied to primary keys, foreign keys and indexes: three calls per table, since none of getPrimaryKeys/getImportedKeys/getIndexInfo accept a pattern. MySQL and Oracle now read the whole schema's constraints in a handful of queries, matching what Postgres already did and what the npm packages already did - verified live against a real 200-table MySQL schema (706ms to 75ms) and a real 65-table Oracle schema (106s to 3.6s), output identical to the per-table path on every shape tried, including composite keys, self-references, and two foreign keys to one table. Approving and running a query used to re-guard text that already carried an injected LIMIT, so the fresh verdict reported no cap and the truncation warning vanished exactly when the reader was looking at the results. The ask-time verdict now survives into execute(). Oracle's dialect notes told the model to write FETCH FIRST while the schema-hint prompt told it not to - the two surfaces had drifted to contradict each other - and the byte-for-byte parity check that would have caught it only ever covered PostgreSQL; it now covers every engine. The rest mirrors the npm-side fix in the same commit: the hint-probe budget shared fairly instead of spent first-come, the pruner no longer capped at a fixed 40 tables regardless of budget, and a truncated list in the prompt - triggers, procedures, sequences, enum values, functions, join paths - now says so instead of presenting a cut-down set as the whole one.
llama-3.3-70b-versatile is retired; every docs page, runnable example and test default naming it as the Groq model to use answered with a 404. Replaced with openai/gpt-oss-20b throughout, the model already named in the store certification notes.
core 0.10.0, sqlite 0.7.0, postgres/mysql/duckdb/oracle 0.5.0, mongodb 0.3.1, jetbrains 0.7.0. server and react are unaffected by this round and stay put.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Started from a real Android sales-force database (186 tables, 1777 columns) that failed to import in
JetBrains with a message reading like a corrupt file. It wasn't - the driver's whole-schema column
call breaches SQLite's own compound-SELECT limit past 500 columns, which any real app database
crosses without trying. That one bug widened into a full pass on schema-scale handling, verified
against real infrastructure end to end.
The headline fix
sqlite-jdbc answers a whole-schema
getColumns()with oneUNION ALLterm per column; SQLiterefuses a compound SELECT past 500 terms. JetBrains now reads columns with
PRAGMA table_info, onetable at a time - what the npm package has always done, which is why VS Code was never affected.
Everything else this uncovered, each verified against real infrastructure
with no column hints at all. Now shared fairly.
budget had room for. Now the budget decides; the cap only guards a pathological schema. A single
wide table no longer evicts every smaller table behind it in the same pass.
to 100 rows reported no warning.
ask-time verdict now survives into execute().
FIRST, the other said don't. The byte-for-byte parity check that should have caught this only ever
covered PostgreSQL; it now covers all five engines.
Postgres and the npm packages already did - verified live: MySQL 706ms→75ms on 200 tables, Oracle
106s→3.6s on 65 tables, output identical to the per-table path on composite keys, self-references,
and multi-FK shapes.
.sqluploads (measured ~68MB heap per 1MB of input - a 60MB file risked an OOM). Both fixed.
so instead of presenting a cut-down set as the whole one.
stopped emitting an unquoted identifier the database would reject.
Verification
Real infrastructure throughout, not mocks: Postgres 18, MySQL 9, MongoDB, a real Oracle container
(gvenzl/oracle-free via Colima), and a real 186-table Android database. Real local Ollama
(
qwen3-coder:30b) end to end against every engine including that real database - all correct,including one case that exercised Oracle's runtime-error recovery path against a genuine model
mistake. The project's own Testcontainers integration suite: 114 tests, 0 failures, across fresh
Postgres/MySQL/Oracle/Mongo containers - including the test proving Oracle's read-only guarantee
holds against a real server.
TS: 3150 pass, 15 skip. Kotlin: 818 unit + 114 integration, 0 failures.
Versions: core 0.10.0, sqlite 0.7.0, postgres/mysql/duckdb/oracle 0.5.0, mongodb 0.3.1, jetbrains
0.7.0. VS Code, browser extension, server and react are unaffected by this round.