Metadata discovery

CHANGELOG.md

@@ -6,6 +6,7 @@
 
 - Added Oxlint lint fence.
 - Document metadata and metadata filtering. Markdown documents can opt into typed metadata through a namespaced frontmatter block (`qmd.metadata` with strings, numbers, booleans, or flat homogeneous arrays), and every search surface — CLI `search`/`vsearch`/`query` via `--filter <json>`, the SDK's `filter` option on `search()`/`searchLex()`/`searchVector()`, the MCP `query` tool, and HTTP `POST /query` and `/search` — accepts one shared recursive filter AST discriminated by `operator`: `and`/`or`/`not` logical groups, `eq`/`ne`/`gt`/`gte`/`lt`/`lte` comparisons, `in`/`nin`/`all` membership, and `exists` presence. Every returned result satisfies the filter (applied before RRF fusion and reranking); like collection filtering, highly selective filters remain best-effort for top-K completeness. Frontmatter stays ordinary searchable content — no chunking, embedding, snippet, or line-number changes — and documents without `qmd.metadata` behave exactly as before. JSON/SDK/MCP/HTTP results now include each document's indexed metadata, and `qmd status` reports how many documents still need metadata extraction (a normal `qmd update` backfills existing indexes).
+- Metadata discovery. Filtering is only useful when the caller knows what to filter on, so every surface now reports the metadata keys, types, and value counts already in the index. The CLI adds `qmd collection metadata [name...]` with `--key <glob>` and `--value <glob>` to select a region of the key/value space (including reverse lookup: which keys hold a value), `--filter <json>` to count only documents matching a filter (same AST as search), and `-n`/`--all`, `--sort count|value`, `--min-count <n>` to shape the per-key value window. `qmd collection list` names each collection's top keys, `qmd collection show` details them with a value preview, and `qmd status` summarizes coverage. The SDK adds `listMetadata(options)`, the MCP server adds a `metadata` tool and lists key names and types per collection in `status`, and the HTTP server adds `POST /metadata`. Counts are documents, not values. Truncation always reports the remainder. Numbers report min, median, and max. Keys whose documents disagree on type (metadata is validated per document, so this can happen within one collection as well as across collections) are split per type with their own document counts rather than resolved. Discovery applies the same extraction gate and scope as filtering, so every value it reports is one an `eq` filter can match. Reads the existing metadata tables, so no re-indexing is needed.
 
 ### Fixed
 

CLAUDE.md

@@ -9,6 +9,7 @@ qmd collection add . --name <n>   # Create/index collection
 qmd collection list               # List all collections with details
 qmd collection remove <name>      # Remove a collection by name
 qmd collection rename <old> <new> # Rename a collection
+qmd collection metadata [name...] # Discover metadata keys, types, and value counts (--key, --value, --filter)
 qmd init                          # Create a project-local .qmd index
 qmd ls [collection[/path]]        # List collections or files in a collection
 qmd context add [path] "text"     # Add context for path (defaults to current dir)
@@ -47,9 +48,15 @@ qmd collection remove mynotes
 # Rename a collection
 qmd collection rename mynotes my-notes
 
-# Show collection details
+# Show collection details, including the top metadata keys
 qmd collection show mynotes
 
+# Discover metadata keys and values to filter on
+qmd collection metadata mynotes
+qmd collection metadata mynotes --key topics
+qmd collection metadata mynotes --value docs-team
+qmd collection metadata mynotes --key topics --filter '{"key":"status","operator":"eq","value":"published"}'
+
 # Set or clear the pre-update hook (runs before re-indexing on `qmd update`)
 qmd collection update-cmd mynotes 'git pull --ff-only'
 qmd collection update-cmd mynotes            # clear

README.md

@@ -152,6 +152,7 @@ runs in a container and a liveness probe connects from a non-loopback address.
 The HTTP server exposes two endpoints:
 - `POST /mcp` — MCP Streamable HTTP (JSON responses, stateless)
 - `POST /query` (alias `/search`) — structured search without the MCP protocol. Accepts the same optional `filter` object as the `query` tool (invalid filters return `400`); see [Metadata Filtering](#metadata-filtering)
+- `POST /metadata` — metadata discovery without the MCP protocol. Same body as the `metadata` tool (invalid filters return `400`); see [Metadata Discovery](#metadata-discovery)
 - `GET /health` — liveness check with uptime
 
 
@@ -198,6 +199,13 @@ Point any MCP client at `http://localhost:8181/mcp` to connect.
 | `get` | `maxLines` | number | Limit returned lines |
 | `get` | `lineNumbers` | boolean | Prefix lines with numbers (default **true**) |
 | `multi_get` | `pattern` | string | Glob pattern or comma-separated list |
+| `metadata` | `collections` | string[] | Restrict discovery to collection names (default: the collections `query` searches) |
+| `metadata` | `key` | string | Glob over metadata key names (`topics` exact, `mem-*` a family) |
+| `metadata` | `value` | string | Glob over values in text form (reverse lookup: which keys hold this value) |
+| `metadata` | `filter` | object | Count only documents matching this filter (same AST as `query`) |
+| `metadata` | `limit` | number | Values shown per key (default 10). `remaining` reports the rest |
+| `metadata` | `sort` | string | `count` (default) or `value` |
+| `metadata` | `minCount` | number | Hide values held by fewer documents (default 1) |
 | `multi_get` | `maxBytes` | number | Skip files larger than N (default 10240) |
 | `multi_get` | `maxLines` | number | Limit lines per file |
 | `multi_get` | `lineNumbers` | boolean | Prefix lines with numbers (default **true**) |
@@ -644,9 +652,14 @@ qmd collection rename myproject my-project
 qmd ls notes
 qmd ls notes/subfolder
 
-# Show collection details (path, glob mask, include status, context count)
+# Show collection details (path, glob mask, include status, context count, top metadata keys)
 qmd collection show notes
 
+# Discover metadata keys, types, and value counts (see Metadata Discovery)
+qmd collection metadata notes
+qmd collection metadata notes --key topics
+qmd collection metadata notes --key topics --filter '{"key":"status","operator":"eq","value":"published"}'
+
 # Include or exclude a collection from default (unscoped) queries
 qmd collection include notes
 qmd collection exclude notes
@@ -985,6 +998,140 @@ Guarantees and limits:
 
 JSON output (`--format json`), the SDK, MCP structured results, and the HTTP endpoints include each result's indexed metadata.
 
+### Metadata Discovery
+
+Filtering is only useful if you know what to filter on. Discovery reports the metadata keys, types, and value counts already in the index, turning "what dimensions exist" into a well-shaped filter in a few steps. It reads the same tables filtering reads: no re-indexing, and every value it reports is one an `eq` filter can match.
+
+The mental model has two halves. `--key` and `--value` are glob patterns (picomatch, as `multi-get` uses) that select **where to look** in the key/value space. `--filter` selects **which documents are counted**. They compose:
+
+| `--key` | `--value` | Question answered |
+|---------|-----------|-------------------|
+| | | Which keys exist, with a window of values each |
+| `topics` | | Everything about one key |
+| `mem-*` | | Which keys look like this |
+| | `docs-team` | Which keys hold this value |
+| | `2025-*` | Which keys hold values shaped like this |
+| `topics` | `type*` | Values of one key matching a pattern |
+
+Start wide and narrow:
+
+```sh
+# Which keys does this collection use? (also shown by `qmd collection show notes`)
+qmd collection metadata notes
+
+# Everything about one key: coverage, distinct count, top values
+qmd collection metadata notes --key topics
+
+# Reverse lookup: which keys hold this value
+qmd collection metadata notes --value docs-team
+
+# What remains after a filter, before committing to it in a query
+qmd collection metadata notes --key topics --filter '{"key":"status","operator":"eq","value":"published"}'
+
+# Then search with the filter you just validated
+qmd query "dependency injection" -c notes --filter '{
+  "operator": "and",
+  "operands": [
+    { "key": "status", "operator": "eq", "value": "published" },
+    { "key": "topics", "operator": "all", "value": ["typescript"] }
+  ]
+}'
+```
+
+The drill-down prints one block per key, in coverage order:
+
+```sh
+qmd collection metadata notes
+```
+
+```
+topics  string[]  388 of 480 documents  1,204 distinct
+  typescript    140
+  sqlite         92
+  search         77
+  architecture   61
+  sqlite-vec     44
+  mcp            39
+  embeddings     35
+  agents         31
+  cli            28
+  testing        26
+1,194 more values, use -n <num> or --all
+
+priority  number  205 of 480 documents  5 distinct
+  min 1  median 3  max 5
+  1 (12)  2 (40)  3 (88)  4 (50)  5 (15)
+
+reviewed  boolean  480 of 480 documents
+  true 61  false 419
+```
+
+A reverse lookup answers "where does this value live" by returning every key that holds it, here a scalar key and an array key:
+
+```sh
+qmd collection metadata notes --value docs-team
+```
+
+```
+owner  string  212 of 480 documents  1 distinct
+  docs-team  212
+
+reviewers  string[]  97 of 480 documents  1 distinct
+  docs-team  97
+```
+
+Options: `-n <num>`/`--all` size the value window (default 10), `--sort count|value` orders it (count descending by default, value ascending for ranges and dates), and `--min-count <n>` drops the long tail. Omitting the collection name covers the default collections, exactly as an unscoped search does.
+
+Rules that matter when reading the output:
+
+- **Counts are documents, not values.** A document with `topics: [a, b]` contributes one to each. Coverage is "documents declaring this key".
+- **Truncation is never silent.** Every capped list ends with the remainder and the flag that removes the cap. Structured results carry it as `remaining`.
+- **Numbers report min, median, and max**, plus the enumerated values when they fit, which is enough to write a sound `gt`/`lt` threshold in one call.
+- **Discovery sees exactly what filtering sees.** Same extraction gate, same active-document rule, same collection scope. Documents still pending extraction are reported on stderr and excluded until `qmd update` runs.
+- **Type conflicts are reported, not resolved.** Metadata is validated one document at a time. Nothing requires two documents to agree on a key's type, whether they sit in the same collection or in different ones, so `priority: 3` in one file and `priority: high` in another both index. Discovery splits such a key by type and gives each type its own document count, which tells you how much of the corpus a typed filter would reach. Within one collection:
+
+```sh
+qmd collection metadata work --key priority
+```
+
+```
+priority  number | string  1,222 of 1,620 documents
+  number     18 docs  min 1  median 2  max 3
+  string  1,204 docs  high (700), medium (380), low (124)
+```
+
+Across collections, each type also names where it comes from:
+
+```sh
+qmd collection metadata --key priority
+```
+
+```
+priority  number | string  1,427 of 2,100 documents
+  number    223 docs  min 1  median 3  max 5               notes, work
+  string  1,204 docs  high (700), medium (380), low (124)  work
+```
+
+`qmd collection list` names each collection's top keys, `qmd collection show <name>` details the top five with a value preview, and `qmd status` summarizes how many keys and files carry metadata.
+
+The same discovery is available on every surface with the same options (`collection`, `key`, `value`, `filter`, `limit`, `sort`, `minCount`) and the same result shape:
+
+```typescript
+// SDK: one flat result, keys in coverage order, each split per type
+const discovery = await store.listMetadata({ collection: "notes", key: "topics", limit: 5 })
+discovery.documents            // active documents in scope (the denominator)
+discovery.keys[0].types[0]     // { type, multiValued, documents, distinctValues, values, remaining, range?, collections }
+
+// Narrowed by a filter: how many documents pass, and what is left to filter on
+const published = await store.listMetadata({
+  collection: "notes",
+  filter: { key: "status", operator: "eq", value: "published" },
+})
+published.filteredDocuments
+```
+
+The MCP `metadata` tool takes the same options with `collections` spelled as on `query`, returns the CLI shape as text and the result as `structuredContent`, and the MCP `status` tool lists each collection's key names and types so an agent's first call reveals that metadata exists. `POST /metadata` accepts the same body as the tool and returns the same result (`400` on an invalid filter).
+
 ### Output Format
 
 Default output is colorized CLI format (respects `NO_COLOR` env).

skills/qmd/SKILL.md

@@ -210,6 +210,21 @@ qmd query "dependency injection" --filter '{"operator":"and","operands":[{"key":
 
 Nodes are discriminated by `operator`: groups `and`/`or` take `operands`, `not` takes one `operand`, and conditions take `key` + `value` with operators `eq`/`ne`/`gt`/`gte`/`lt`/`lte` (comparison), `in`/`nin`/`all` (membership), or `exists` (presence). Matching is typed and exact; missing keys do not match `ne`/`nin` (add an `exists: false` branch in an `or` group to include them). The MCP `query` tool accepts the same AST as a `filter` object. JSON output includes each result's `metadata`.
 
+## Discover metadata before filtering
+
+Do not guess keys or values. `qmd collection show <name>` lists the top keys with types and a value preview, and `qmd collection metadata` drills in. `--key` and `--value` are globs that pick where to look, and `--filter` picks which documents are counted:
+
+```bash
+qmd collection metadata notes                          # every key, ten values each
+qmd collection metadata notes --key topics             # one key: coverage, distinct count, top values
+qmd collection metadata notes --value docs-team        # reverse lookup: which keys hold this value
+qmd collection metadata notes --key topics --filter '{"key":"status","operator":"eq","value":"published"}'
+```
+
+Read the header first: `topics  string[]  388 of 480 documents  1,204 distinct` tells you coverage and cardinality before you commit to a filter, and with `--filter` it tells you how many documents pass. Counts are documents, not values. A `N more values, use -n <num> or --all` footer means the list was cut. Raise `-n` rather than assuming the rest. Numbers print `min`, `median`, and `max` so you can write a `gt`/`lt` threshold in one call. A key shown as `number | string` means documents disagree on type. Metadata is validated per document, so this happens within a single collection as readily as across collections. Each type reports its own document count. Filter by the type that covers the documents you want. Every value shown can be matched with `eq` under the same collection scope.
+
+Over MCP, call the `metadata` tool (same options, `collections` as an array) and read `remaining` and `range` from the structured result. The `status` tool lists each collection's key names and types, so check it first.
+
 ## MCP Tool: `query`
 
 When using the MCP server, prefer structured searches: