Metadata discovery

[aaronccasanova]

Follow-up to tobi#910. That PR let documents declare typed metadata and every search surface filter on it. This one closes the loop: every surface can now report the metadata keys, types, and value counts already in the index, so a filter can be written from what is indexed instead of guessed.

qmd collection metadata notes

status  string  480 of 480 documents  3 distinct
  published  312
  draft      141
  archived    27

topics  string[]  388 of 480 documents  1,204 distinct
  typescript    140
  sqlite         92
  search         77
  architecture   61
  ...
1,194 more values, use -n <num> or --all

priority  number  205 of 480 documents  5 distinct
  min 1  median 3  max 5
  1 (12)  2 (40)  3 (88)  4 (50)  5 (15)

Every value shown is one a filter will match, and the counts say how many documents each condition reaches:

qmd query "dependency injection" -c notes --filter '{
  "operator": "and",
  "operands": [
    { "key": "status", "operator": "eq", "value": "published" },
    { "key": "topics", "operator": "all", "value": ["typescript"] },
    { "key": "priority", "operator": "gte", "value": 3 }
  ]
}'

The same discovery is available on the SDK (listMetadata()), the MCP server (a metadata tool, plus key names in status), and HTTP (POST /metadata), with the same options and the same result shape. It reads the tables tobi#910 created, so there is no re-indexing, no schema change, and no new dependency.

The motivating use case is the same one that motivated filtering: agentic retrieval. A filter is only useful if the caller knows which keys exist and what values they hold, and an agent working against an index it did not build has no way to learn that today except guessing or reading files one by one. Discovery gives it a first call that reveals the dimensions and a second call that validates a filter before it ever runs a search.

What this adds

• qmd collection metadata [name...], a drill-down with --key <glob> and --value <glob> to select a region of the key/value space, --filter <json> to count only documents matching a filter, and -n/--all, --sort count|value, --min-count <n> to shape the value window per key.
• Metadata at a glance in the places people already look: qmd collection list names each collection's top keys, qmd collection show details them with a value preview, and qmd status summarizes coverage.
• listMetadata(options) on the SDK, a metadata tool on the MCP server, POST /metadata on the HTTP server, and per-collection key names and types in the MCP status tool so an agent's first call already reveals that metadata exists.
• One store function (listMetadata() in src/metadata-store.ts) and one renderer (src/metadata-format.ts) behind every surface, so the CLI and the MCP tool print the same text and every structured surface returns the same object.

What this deliberately does not change

• Nothing is indexed and nothing is migrated. Discovery is a read over document_metadata_values and its existing covering indexes.
• The filter language is untouched. Discovery consumes the same AST, compiled by the same compiler, under the same eligibility gate, so what discovery reports and what filtering matches cannot drift.
• No new dialect. Key and value patterns are picomatch globs, the dialect multi-get already uses.
• No --format json on collection commands (deferred in Add metadata support tobi/qmd#910, still deferred). Structured output lives on the SDK, MCP, and HTTP.
• Nothing changes for anyone who never writes qmd.metadata.

The mental model

Two globs select where to look, and a filter selects which documents are counted. They compose:

--key	--value	Question answered
		Which keys exist, with a window of values each
topics		Everything about one key
mem-*		Which keys look like this
	docs-team	Which keys hold this value
	2025-*	Which keys hold values shaped like this
topics	type*	Values of one key matching a pattern

Adding --filter to any row shows what remains after narrowing. The header then reports how many documents pass, which is the number you want before committing to that filter in a query:

qmd collection metadata notes --key topics --filter '{"key":"status","operator":"eq","value":"published"}'

topics  string[]  260 of 480 documents match filter  811 distinct
  typescript    104
  sqlite         70
  search         58
  ...
806 more values, use -n <num> or --all

A reverse lookup returns every key that holds a value, here a scalar key and an array key:

qmd collection metadata notes --value docs-team

owner  string  212 of 480 documents  1 distinct
  docs-team  212

reviewers  string[]  97 of 480 documents  1 distinct
  docs-team  97

Reading the output

Four rules, each observable in tests:

• Counts are documents, not values. A document with topics: [a, b] contributes one to each. Coverage is "documents declaring this key".
• Truncation is never silent. Every capped list ends with the exact remainder and the flag that removes the cap. Structured results carry it as remaining, never left for the caller to infer.
• Numbers report min, median, and max, plus the enumerated values when they fit, which is enough to write a sound gt/lt threshold in one call.
• Type conflicts are reported, not resolved. Metadata is validated one document at a time, and nothing requires two documents to agree on a key's type, whether they sit in the same collection or in different ones. A key that holds numbers in some files and labels in others splits by type, each with its own document count, so you can see how much of the corpus a typed filter would reach:

qmd collection metadata work --key priority

priority  number | string  1,222 of 1,620 documents
  number     18 docs  min 1  median 2  max 3
  string  1,204 docs  high (700), medium (380), low (124)

Across collections, each type also names where it comes from:

qmd collection metadata --key priority

priority  number | string  1,427 of 2,100 documents
  number    223 docs  min 1  median 3  max 5               notes, work
  string  1,204 docs  high (700), medium (380), low (124)  work

Design notes

The output shapes came out of a survey of how Elasticsearch terms aggregations, Weaviate aggregate queries, and Postgres pg_stats report cardinality, truncation, and distributions. The retrieval side reuses tobi#910's machinery rather than adding a parallel path. Details collapsed below.

One options object and one result on every surface

interface ListMetadataOptions {
  collection?: string | string[];   // CLI positional, MCP/HTTP `collections`
  key?: string;                     // picomatch over key names
  value?: string;                   // picomatch over values in text form
  filter?: MetadataFilter;          // same AST as search
  limit?: number;                   // values per key, default 10
  sort?: "count" | "value";
  minCount?: number;
}

interface ListMetadataResult {
  documents: number;                // active documents in scope, the denominator
  filteredDocuments?: number;       // present only when a filter was given
  keys: MetadataKeySummary[];       // coverage order
}

interface MetadataKeySummary {
  key: string;
  documents: number;
  types: MetadataKeyTypeSummary[];  // length > 1 is a type conflict
}

interface MetadataKeyTypeSummary {
  type: "string" | "number" | "boolean";
  multiValued: boolean;             // some document holds this key as an array
  documents: number;
  distinctValues: number;
  values: { value: string | number | boolean; documents: number }[];
  remaining: number;                // distinct values not shown, exact
  range?: { min: number; median: number; max: number };
  collections: string[];
}

The result carries its own denominator so the "N of M documents" header can be computed on every surface without a second call, and so the HTTP route can return the object as its own envelope, identical to the MCP structuredContent. Scope resolution is the caller's job, mirroring search: the CLI resolves omitted names to the default collections, MCP and HTTP fall back to their existing default collection list, and the SDK passes what it was given.

minCount scopes only values, distinctValues, and remaining. Per-type documents, multiValued, range, and collections describe every matching value so coverage stays honest when the long tail is hidden. Median is computed over value rows (each element of a number[] counts), following Weaviate's aggregate semantics.

Value patterns at high cardinality

The motivating corpus for --value is an agent memory collection with a very large distinct-value set, where post-filtering every value in JavaScript is the wrong default. picomatch is the only dialect the user sees, but when a pattern contains only literals, *, and ? (none of \ ( ) [ ] { } | ! + @, no **, no ./), it is also a valid SQLite GLOB whose match set is a provable superset of the picomatch match set. String candidates are pruned with that GLOB inside the query and picomatch confirms every survivor, so the pushdown can only ever remove rows picomatch would have rejected anyway. Numbers and booleans skip the prefilter because SQLite renders REALs differently from String(number). Patterns needing braces, extglobs, or negation fall back to picomatch alone. Tests assert the superset property on the pushed subset and the fallback on the rest.

Keys are matched in JavaScript over the small distinct key set, and both matched keys and matched values bind into the aggregates as one JSON parameter (json_each and a MATERIALIZED CTE), so the SQLite variable limit never applies.

Same gate, same scope, same compiler

Discovery builds one eligible-documents CTE with exactly the predicate filtered search uses (active = 1, current extraction_version, no extraction_error), the optional collection scope, and the optional compiled filter from compileMetadataFilter(). Every aggregate joins document_metadata_values to that CTE. This is why the PR can make the guarantee that every value discovery reports is matchable with eq under the same collections: a test asserts it by running searchFTS with an eq filter on each reported value. Documents still pending extraction are counted in the denominator, excluded from the aggregates, and reported on stderr the way filtered search already reports them.

The status views (collection list, collection show, qmd status, the MCP status tool) use a lighter listMetadataKeys() (one GROUP BY key, value_type, no values) so a first look stays cheap. Only collection show calls listMetadata(), with limit: 3, because its rows preview values.

Performance

On a synthetic index of 10,000 documents carrying 60,000 metadata value rows, listMetadata() runs in 90-250 ms per call depending on options, served by the covering partial indexes tobi#910 added. No new indexes were needed and no query plan work was done. The status views are a single GROUP BY over the same indexes.

Testing

64 new tests across three suites, green under both Node (vitest) and Bun:

• Store (test/metadata-discovery.test.ts, 39): counting, the extraction gate, scope and filter narrowing, key and value patterns including the GLOB superset property, remaining, sorts, minCount, ranges and medians, type splits within and across collections, and the discovery/filtering agreement test.
• CLI (test/metadata-cli.test.ts, 15): every flag through a spawned qmd, byte-exact rendering of each key type and the type split, the list/show/status lines, and exit codes and messages for every invalid input.
• Surfaces (test/metadata-surfaces.test.ts, 10): the SDK method, getStatus() keys, the MCP metadata and status tools, and POST /metadata including every 400 path.

Full suite: typecheck, lint, 1310 Node tests, 1310 Bun tests, and the package smoke all pass. The five commits build in dependency order (store, CLI drill-down, status views, SDK/MCP/HTTP, docs) and read best one at a time.

Limitations and future work

• Numbers report min, median, max, and an enumeration when it fits. There is no histogram or percentile output yet. If a corpus needs one, range is the place to grow it compatibly.
• Dates are strings, so --value '2025-*' finds them but nothing orders or ranges them as dates. That follows Add metadata support tobi/qmd#910's first-version restriction and can be relaxed there first.
• multiValued means "some document holds more than one value". A key whose every array has one element is indistinguishable from a scalar key at the storage level and reports as scalar.
• --format json on collection commands remains deferred. The structured result is available today from the SDK, MCP, and HTTP.

Related work

• Add metadata support tobi/qmd#910: metadata and metadata filtering. This PR reads the tables and reuses the filter compiler and eligibility gate it introduced, and changes none of its behavior.