Metadata discovery

[aaronccasanova]

Follow-up to #910, on top of #956. That PR let documents declare typed metadata and every search surface filter on it. This one closes the loop: every surface can now report the metadata keys, types, and value counts already in the index, so a filter can be written from what is indexed instead of guessed.

qmd collection metadata notes

status  string  480 of 480 documents  3 distinct
  published  312
  draft      141
  archived    27

topics  string[]  388 of 480 documents  1,204 distinct
  typescript    140
  sqlite         92
  search         77
  architecture   61
  ...
1,194 more values, use --value-limit <n>, --value-offset <n>, or --all-values

priority  number  205 of 480 documents  5 distinct
  min 1  median 3  max 5
  1 (12)  2 (40)  3 (88)  4 (50)  5 (15)

Every value shown is one a filter will match, and the counts say how many documents each condition reaches:

qmd query "dependency injection" -c notes --filter '{
  "operator": "and",
  "operands": [
    { "field": "status", "operator": "eq", "value": "published" },
    { "field": "topics", "operator": "all", "value": ["typescript"] },
    { "field": "priority", "operator": "gte", "value": 3 }
  ]
}'

Narrowing the report uses the filter language itself. --match takes the same AST as --filter, evaluated against each metadata entry, so an agent that has learned one has learned the other:

qmd collection metadata notes --match '{
  "operator": "and",
  "operands": [
    { "field": "key", "operator": "eq", "value": "priority" },
    { "field": "value", "operator": "gte", "value": 3 }
  ]
}'

priority  number  153 of 480 documents  3 distinct
  min 3  median 3  max 5
  3 (88)  4 (50)  5 (15)

The same discovery is available on the SDK (listMetadata()), the MCP server (a metadata tool, plus key names in status), and HTTP (POST /metadata), with the same options and the same result shape. It reads the tables #910 created, so there is no re-indexing, no schema change, and no new dependency.

The motivating use case is the same one that motivated filtering: agentic retrieval. A filter is only useful if the caller knows which keys exist and what values they hold, and an agent working against an index it did not build has no way to learn that today except guessing or reading files one by one. Discovery gives it a first call that reveals the dimensions and a second call that validates a filter before it ever runs a search.

What this adds

• Text and type operators on every surface that accepts a filter: contains, prefix, and suffix for substring matching over string values, type to match the stored type of a key's values, and an optional caseInsensitive flag on any condition whose value is a string.
• qmd collection metadata [name...], a drill-down with --filter <json> to count only documents matching a filter, --match <json> to select which metadata entries are reported (same AST, evaluated per entry), two windows that page (--key-limit/--key-offset/--all-keys over keys, --value-limit/--value-offset/--all-values over the values of each key and type), and --sort count|value, --min-count <n> to order and trim values.
• Metadata at a glance in the places people already look: qmd collection list names each collection's top keys, qmd collection show details them with a value preview, and qmd status summarizes coverage.
• listMetadata(options) on the SDK, a metadata tool on the MCP server, POST /metadata on the HTTP server, and per-collection key names and types in the MCP status tool so an agent's first call already reveals that metadata exists.
• One store function (listMetadata() in src/metadata-store.ts) and one renderer (src/metadata-format.ts) behind every surface, so the CLI and the MCP tool print the same text and every structured surface returns the same object.

What this deliberately does not change

• Nothing is indexed and nothing is migrated. Discovery is a read over document_metadata_values and its existing covering indexes.
• Existing filter semantics are preserved. Discovery uses the same grammar and eligibility gate, with independent tests comparing both compiler scopes and entry matches against their expected results.
• No second language. There is one grammar, applied to two kinds of record.
• No --format json on collection commands (deferred in Add metadata support #910, still deferred). Structured output lives on the SDK, MCP, and HTTP.
• Metadata remains opt-in. Documents without qmd.metadata keep their existing indexing and unfiltered-search behavior.

The mental model

The filter AST is a predicate grammar. A condition tests one field of the record under evaluation against value using operator, and nothing in that grammar is specific to documents. Discovery applies it to two kinds of record:

Flag	Record evaluated	Its fields	A condition's field names
--filter	A document	Its metadata keys (status, topics, priority, ...)	One of those keys
--match	A metadata entry	key, value	One of those two

Same metadata, different unit. --filter narrows documents by their metadata. --match narrows the metadata itself. Every operator applies to both, including type, the text operators, caseInsensitive, and and/or/not. The only conditions a match rejects are exists and all, which have no meaning for a single entry.

--match	Question answered
	Which keys exist, with a window of values each
{"field":"key","operator":"eq","value":"topics"}	Everything about one key
{"field":"key","operator":"prefix","value":"mem-"}	A family of keys
{"field":"key","operator":"in","value":["tags","topics","labels"]}	Which of these key names exist
{"field":"value","operator":"eq","value":"docs-team"}	Which keys hold this value
{"field":"value","operator":"prefix","value":"2025-"}	Which keys hold values shaped like this
{"field":"value","operator":"type","value":"boolean"}	Which keys hold booleans
and of key eq priority and value gte 3	Values of one key above a threshold
and of key eq priority and value type number	The numeric side of a key whose documents disagree on type

Because the result is per key and per type, a composed match returns a mixture in one call: several keys, several value regions, or a condition joined across the two fields.

Adding --filter to any row shows what remains after narrowing. The output opens with how many documents pass, and every coverage count is measured against that population, which is the number you want before committing to that filter in a query:

qmd collection metadata notes \
  --match '{"field":"key","operator":"eq","value":"topics"}' \
  --filter '{"field":"status","operator":"eq","value":"published"}' \
  --value-limit 5

filter: 312 of 480 documents

topics  string[]  260 of 312 documents  811 distinct
  typescript    104
  sqlite         70
  search         58
  architecture   40
  mcp            31
806 more values, use --value-limit <n>, --value-offset <n>, or --all-values

A reverse lookup returns every key that holds a value, here a scalar key and an array key:

qmd collection metadata notes --match '{"field":"value","operator":"eq","value":"docs-team"}'

owner  string  212 of 480 documents  1 distinct
  docs-team  212

reviewers  string[]  97 of 480 documents  1 distinct
  docs-team  97

Reading the output

Rules observable in tests:

• Counts are documents, not values. A document with topics: [a, b] contributes one to each. Coverage is "documents declaring this key", out of the documents the filter admits when there is one. The filter: line prints even when the filtered documents declare no metadata.
• A bare string is exactly the value. A string prints as-is only when nothing else could be read from it, and otherwise as an escaped JSON string ("42" beside a number 42, "a (1), b" in a compact list). Bare strings still need quotes in a JSON operand; quoted ones are the operand.
• Truncation is never silent, and nothing is unreachable. Both the key list and every value list are windowed in SQL with a default (50 keys, 10 values), and both page with an offset. Every windowed list ends with the exact remainder and the flags that reach it. Structured results carry totalKeys/remainingKeys and distinctValues/remainingValues, never left for the caller to infer.
• Numbers report min, median, and max, plus the enumerated values when they fit, which is enough to write a sound gt/lt threshold in one call.
• Type conflicts are reported, not resolved. Metadata is validated one document at a time, and nothing requires two documents to agree on a key's type, whether they sit in the same collection or in different ones. A key that holds numbers in some files and labels in others splits by type, each with its own document count, so you can see how much of the corpus a typed filter would reach:

qmd collection metadata work --match '{"field":"key","operator":"eq","value":"priority"}'

priority  number | string  1,222 of 1,620 documents
  number     18 docs  min 1  median 2  max 3
  string  1,204 docs  high (700), medium (380), low (124)

Across collections, each type also names where it comes from:

qmd collection metadata --match '{"field":"key","operator":"eq","value":"priority"}'

priority  number | string  1,427 of 2,100 documents
  number    223 docs  min 1  median 3  max 5               notes, work
  string  1,204 docs  high (700), medium (380), low (124)  work

Adding {"field":"value","operator":"type","value":"number"} to the match reports the numeric side alone, and {"field":"priority","operator":"type","value":"number"} is the filter that reaches exactly those documents.

Design notes

The output shapes came out of a survey of how Elasticsearch terms aggregations, Weaviate aggregate queries, and Postgres pg_stats report cardinality, truncation, and distributions. The retrieval side reuses #910's machinery rather than adding a parallel path. Details collapsed below.

One options object and one result on every surface

interface ListMetadataOptions {
  collection?: string | string[];   // CLI positional, MCP/HTTP `collections`
  match?: MetadataMatch;            // same grammar, evaluated per metadata entry
  filter?: MetadataFilter;          // same AST as search
  keyLimit?: number;                // keys reported, default 50; Infinity removes the window
  keyOffset?: number;               // keys skipped, in report order
  valueLimit?: number;              // values per key and type, default 10; Infinity removes the window
  valueOffset?: number;             // values skipped per key and type, in `sort` order
  sort?: "count" | "value";
  minCount?: number;
}

interface ListMetadataResult {
  documents: number;                // active documents in scope
  filteredDocuments?: number;       // present only when a filter was given; then the coverage denominator
  totalKeys: number;                // keys with a matching entry, before the window
  keys: MetadataKeySummary[];       // the window, coverage order
  remainingKeys: number;            // keys after the window, exact
}

interface MetadataKeySummary {
  key: string;
  documents: number;
  types: MetadataKeyTypeSummary[];  // length > 1 is a type conflict
}

interface MetadataKeyTypeSummary {
  type: "string" | "number" | "boolean";
  multiValued: boolean;             // some document holds this key as an array
  documents: number;
  distinctValues: number;
  values: { value: string | number | boolean; documents: number }[];
  remainingValues: number;          // distinct values after the window, exact
  range?: { min: number; median: number; max: number };
  collections: string[];
}

The result carries its own denominator so the "N of M documents" header can be computed on every surface without a second call, and so the HTTP route can return the object as its own envelope, identical to the MCP structuredContent. Scope resolution is the caller's job, mirroring search: the CLI resolves omitted names to the default collections, MCP and HTTP fall back to their existing default collection list, and the SDK passes what it was given.

The option names are the same on every surface (--key-limit on the CLI is keyLimit on the SDK, MCP, and HTTP), so an agent that learns one learns them all. Every option is validated at the store boundary and a value outside its domain throws MetadataOptionError naming the option (Invalid keyLimit: expected a positive integer or Infinity, received 0). The CLI and MCP reject bad values up front with their own messages, HTTP returns the store's message as a 400, and the SDK throws it, so no surface silently clamps.

minCount scopes only values, distinctValues, and remainingValues. Per-type documents, multiValued, range, and collections describe every matching value so coverage stays honest when the long tail is hidden. Median is computed over value rows (each element of a number[] counts), following Weaviate's aggregate semantics.

One grammar, two record types

src/metadata-filter.ts keeps one validator and one grammar. MetadataPredicate<Condition> is the recursive shape (a condition, or and/or/not over predicates), and the two public types instantiate it with the conditions each record admits: MetadataFilter = MetadataPredicate<MetadataCondition> over a document (any field, every operator) and MetadataMatch = MetadataPredicate<MetadataEntryCondition> over an entry (field is "key" | "value", without exists and all, which speak about the set of values a document holds). So a document-only condition in a match is a TypeScript error for SDK users and a runtime error for everyone else, and the two cannot drift because they are one type. A vitest typecheck file (test/metadata-filter.test-d.ts) pins the shape.

parseMetadataFilter() and parseMetadataMatch() share every rule and limit and differ in a record-type parameter: for an entry, a condition's field must be key or value, and exists and all are rejected with a message that says why. Errors name the failing JSON path in both, prefixed Invalid metadata filter at or Invalid metadata match at.

Compilation differs by target. compileMetadataFilter() uses candidate-local EXISTS probes for search and uncorrelated document-ID sets for discovery. compileMetadataMatch() emits a predicate over one document_metadata_values row: the value field resolves to the typed columns guarded by value_type, and the key field resolves to the key column with the type fixed to 'string', so a number or boolean operand against it fails its guard and matches nothing, the same outcome a type mismatch has in a filter. Both compilers share the operand-to-column logic, with independent boolean evaluation testing their typed semantics.

Every operator has an exact SQL form and nothing is re-checked in JavaScript. contains uses instr(). prefix and suffix compare UTF-8 bytes (substr(CAST(col AS BLOB), ...)) with the operand's byte length bound from JavaScript, because SQLite's text length() and substr() stop at an embedded NUL and the stored string domain admits one. UTF-8 is self-synchronizing, so a byte-prefix of a whole operand is exactly a character-prefix. COALESCE(comparison, 0) makes an empty BLOB substring a false comparison, not SQL NULL, so negation includes the empty string when it should. caseInsensitive folds ASCII on both sides: lower() on the column and the same A-Z range on the operand, so the two agree exactly and non-ASCII letters compare exactly. The match predicate lives in the JOIN of the aggregation queries, so a high-cardinality key never round-trips its values, and the matched key set for the array-ness pass binds as one JSON parameter through json_each.

Same gate, same scope, same compiler

Discovery builds one eligible-documents CTE with exactly the predicate filtered search uses (active = 1, current extraction_version, no extraction_error), the optional collection scope, and the optional filter from the same compileMetadataFilter() search calls. Every aggregate joins document_metadata_values to that CTE. This is why the PR can make the guarantee that every value discovery reports is matchable with eq under the same collections: a test asserts it by running searchFTS with an eq filter on each reported value. Documents still pending extraction are counted in the denominator, excluded from the aggregates, and reported on stderr the way filtered search already reports them, scoped to the collections the command reads.

Status views use dedicated queries. collection list and SDK/MCP status batch the collection overviews, including vocabulary totals, in one query over ordinal-zero entries (one row per document/key). CLI status uses key counts and document-existence probes. collection show requests five keys and three values through listMetadata(). MCP initialization reads only the index facts its instructions use, without computing metadata summaries. The public status result still carries each collection's ten most covered keys and metadataKeyCount.

Performance

Discovery reuses key selection and value-count aggregation within each report. Collection overviews are batched, and MCP initialization does not compute unused metadata summaries. The filter compiler also avoids repeated broad range scans in candidate filtering. These changes need no new index, cache, or re-indexing.

The windows bound key/value lists, not query work or contributing collection lists. Exact grouping, medians, ordering, and filter evaluation remain corpus-dependent.

Query work, measurements, and trade-offs

Filter access. Discovery compiles each document condition to an uncorrelated ID set. Candidate filtering keeps covering equality probes and guides range, presence, type, text, and folded-equality conditions toward document-local rows. Plan tests cover both statistics states. A corpus-scope set may read outside the final collection scope, so this is not a guarantee of work proportional to the returned region.

Shared report work. The selected-key ranking is materialized within the type-statistics statement, avoiding the small-window coroutine plan. The report then reuses those key names rather than rebuilding the ranking in each aggregate. Collection provenance is read with type statistics, then sorted by UTF-8 bytes without requiring SQLite 3.44's aggregate ordering syntax. Value counts, distinct totals, and pagination share one grouped query. A total-carrier row preserves exact counts past the end of a value window.

Status and initialization. One grouped overview query computes all collections' key coverage, types, and totals from ordinal-zero entries. MCP initialization uses a separate internal index-summary path, so every stateless HTTP request no longer pays for metadata summaries it discards. The explicit status tool still computes and returns current metadata summaries.

Author measurements on macOS arm64: 60,000 documents, 522,404 metadata rows, 47 keys, six collections, with arrays and type conflicts. Same seeded fixture and commands for 0cc9856 and f32848b. Each number is a median of three calls after a warmup, without ANALYZE. These are workload measurements, not capacity or complexity guarantees.

Store call	Bun before	Bun after	Node before	Node after
Default discovery	5,767 ms	2,729 ms	5,394 ms	2,609 ms
Collection show shape (5 keys, 3 values)	1,064 ms	473 ms	770 ms	407 ms
One-key discovery	1,532 ms	730 ms	1,334 ms	669 ms
Range-filtered discovery	3,580 ms	2,260 ms	2,769 ms	1,988 ms

Bun 1.3.6 uses SQLite 3.51.2. Node 24.15.0 uses better-sqlite3's SQLite 3.53.4. Process-startup costs are excluded above and included in these separate Bun CLI measurements:

Public path	Before	After
CLI status	452 ms	375 ms
CLI collection list	1,204 ms	606 ms
CLI collection show	1,085 ms	614 ms
MCP tools/list, including per-request initialization	1,403 ms	108 ms
MCP status, including per-request initialization	2,861 ms	666 ms

The deliberate trade-off is fresh, exact metadata summaries without maintained aggregate state. An explicit SDK status call still costs about 0.5-0.7 seconds on this fixture, compared with roughly 74 ms for main's status without metadata summaries. Initialization avoids that additional work. A covering-index experiment gave a modest overview improvement, while a canonical-JSON implementation was slower, so neither was adopted.

Sorting and temporary B-trees remain, and a long read snapshot can retain WAL history. There is no claim of universal linear scaling, constant memory, or cost independent of the index. Broader workload tuning remains useful follow-up work.

Exact under concurrency and at the edges of the number line

• One snapshot. The whole report runs inside one deferred read transaction. Under WAL another connection can commit while the report reads, and its counts still come from one state. A test commits a second connection's write between the range statement and the median statement and checks the result is the original distribution (min 1, median 1, max 1), not the min 1, median 100, max 1 it produced before.
• Medians. The even-count midpoint is taken in JavaScript, (lo + hi) / 2 when the sum is finite and lo / 2 + hi / 2 in the overflow region, so it is finite and correctly rounded across the double range, adjacent subnormals included. SQL's AVG sums first and returned Infinity for 1e308 and 1.5e308.
• Window integers. Options accept safe integers only (1e30 passes Number.isInteger and then fails LIMIT ? with a datatype mismatch). Both drivers can bind safe integers as REAL, so the production value-window predicate casts before summing as INTEGER. The regression executes the emitted predicate and its bindings at Number.MAX_SAFE_INTEGER + 2, where JavaScript addition would round.
• Binding budget. The widest planned statement is checked against METADATA_SQL_BINDING_BUDGET (30,000, under SQLite's default 32,766) before discovery queries are prepared. The calculation covers key selection, where the match binds twice, and value aggregation, including the selected-key parameter and window operands. MetadataBindingBudgetError surfaces as a CLI exit, an MCP tool error, and an HTTP 400.
• Vector lookup bindings. Exact scans and the capped global fallback bind candidate IDs as one JSON list during document lookup. Candidate count no longer consumes the headroom a parser-valid metadata filter needs under Node's variable limit. The filter is still applied to each document, including nonmatching paths that share content with a matching one.
• Strings print as themselves. A string prints bare only when nothing else could be read from it. Empty, padded, containing a quote, backslash, control character, line separator, or a delimiter of the compact value (count), ... list, or reading as a JSON literal or the list's remainder tail, it prints as a JSON string with every control escaped (DEL, C1, U+2028/9 included, which JSON.stringify leaves literal). One rule for both layouts, so "a (1), b" (1) is one value and never two, and ordinary values cost nothing.

Testing

132 added tests across seven suites. Runtime tests run under Node and Bun, with five additional compile-time assertions through Vitest.

Coverage and verification

• Filter language (test/metadata-filter.test.ts, +12): the new operators and caseInsensitive through validation and SQL semantics (type mismatch never matches, array any-element semantics, whole-character prefix and suffix, the whole value past an embedded NUL including self-prefix and self-suffix, ASCII-only folding), every semantic case in both filter scopes, the plan of every operator with and without statistics, the corpus scope's SQL and plan, and the match parser and compiler.
• Predicate types (test/metadata-filter.test-d.ts, 5): compile-time assertions, run by vitest's typecheck pass, that MetadataMatch composes like a filter, admits only key and value, rejects exists and all, and that MetadataFilter still admits every document condition.
• Search (test/metadata-search.test.ts, +1): a model-free, near-ceiling filter across the 20,000-vector exact/fallback boundary, including exclusion of a nonmatching path sharing the content hash.
• Store (test/metadata-discovery.test.ts, 66): counting, the extraction gate, scope and filter narrowing, the match over both fields (text, membership, typed comparison, type selection, negation, cross-field or, case folding, validation), both windows with offsets, exact remainders past the end, the window applied to matched keys, pages as prefixes of the full result under one collation on both readers, the materialized key window and document-set filter at the five- and ten-key sizes with and without statistics, snapshot isolation against a writer that commits between the range and the median, option validation including unsafe integers and the largest safe ones, the actual emitted window predicate and bindings evaluated past 2^53, the binding budget (each predicate alone under it, together over it), sorts, minCount, ranges and medians across the double range, type splits within and across collections, bounded key overviews through lifecycle changes, batched status and initialization work guards, independent empty-string/composition expectations, the discovery/filtering agreement test, and UTF-8 collection provenance without SQLite 3.44-only syntax.
• CLI (test/metadata-cli.test.ts, 24): every flag through a spawned qmd, byte-exact rendering of each key type, the type split, the filter line with its denominator including over a result with no metadata, paging both windows, a composed match, type selection, the list/show/status lines, the pending-extraction warning scoped to the collections read, and exit codes and messages for invalid-input cases, including unsafe integers, unsupported formats, and the search flags -n and --all.
• Renderer (test/metadata-format.test.ts, 8): string identity (every ambiguous form quoted and round-tripping through JSON.parse, plain strings bare, the compact list read back honoring quotes), no raw control character in the output, numbers and booleans bare, the empty states under a filter, and a POSIX shell round trip proving apostrophes and substitution syntax in a key remain literal data in a drill-down hint.
• Surfaces (test/metadata-surfaces.test.ts, 16): the SDK method with windows, option errors, unsafe integers, and the binding budget, getStatus() keys and counts, the MCP metadata tool (paging, the filter line over an empty result, the budget as a tool error) and status tool, and POST /metadata including the documented 400 paths. Empty-string negation is exercised through SDK, HTTP, and MCP. A real MCP tools/list request succeeds while metadata value-table reads are unavailable in its isolated fixture.

The empty-string, production endpoint, repeated key-ranking, MCP initialization, and shell-hint regressions each fail without their correction. An author rerun of the auditor's independent evaluator passes 14,276 predicate comparisons and 120 complete aggregate comparisons per runtime, without the experimental SQL workaround. Full suite: typecheck, lint, 1378 Node tests, 1373 Bun tests, and the package smoke all pass. The 105 filter, discovery, and vector tests also pass under SQLite 3.43.2 through Bun's custom-library path. The new vector regression fails before the candidate-ID correction, and the real discovery query fails on SQLite 3.43 before the ordering correction.

The final author close-out also rechecks exact 29,999/30,000/30,001 binding boundaries, live-writer interleavings, nested transactions and cleanup, independently expected lifecycle overviews, and an SDK workflow from empty metadata through annotation, discovery, filtering, and extraction-error recovery. These author checks are not represented as an independent review of the final head.

Trade-offs and future work

• Numbers report min, median, max, and an enumeration when it fits. There is no histogram or percentile output yet. If a corpus needs one, range is the place to grow it compatibly.
• Dates are strings, so a prefix match on 2025- finds them and --sort value orders them, but nothing ranges them as dates. That follows Add metadata support #910's first-version restriction and can be relaxed there first.
• caseInsensitive folds ASCII letters. Full Unicode folding needs a folded column or an ICU build of SQLite, either of which is a storage change.
• There is no regular-expression operator. The text operators cover substring, prefix, and suffix. If a need appears, the operator would be named for intent with one declared flavor, in both record types.
• multiValued means "some document holds more than one value". A key whose every array has one element is indistinguishable from a scalar key at the storage level and reports as scalar.
• Collection commands print text and reject unsupported format flags. Structured results are available from the SDK, MCP, and HTTP.

Related work

• Name the filter condition's target field #956: names the condition's target field. This PR is based on it, and the two-record framing above is why that name is the right one.
• Add metadata support #910: metadata and metadata filtering. This PR reads the tables and reuses the filter compiler and eligibility gate it introduced. It extends the filter grammar, keeps every existing semantic, and changes how the compiler's conditions reach rows (see Performance), which query and search inherit. It also prevents vector candidate IDs from exhausting the remaining SQL bindings when a large metadata filter is applied.