Answer a question in English against a corpus

[skearnes]

Summary

ord_schema.search.nl turns a question into a Query, runs it against a Corpus, and says what came back:

answer = nl.ask("which reactions use pyridine as a solvent?", corpus)
answer.query   # the Query that ran
answer.table   # what the search returned
answer.text    # "24,930 reaction records that used pyridine as a solvent..."

Implements tasks 2–6 of the plan in ord-logbook#45. The measurements behind every design choice are in the entry beside it.

A model cannot be constrained to emit a valid Query: structured outputs and strict tools share a validator that refuses circular references, then refuses what is left once the recursion is removed as too large. So translation is generation checked afterwards — a coercing parse, pydantic, the compiler, and one repair turn carrying the compiler's own "did you mean".

A bug this turned up

The first real question I asked returned zero rows over a corpus holding 24,930 matches. The default resolver asks PubChem, which returns pyridine as C1=CC=NC=C1; the projection stores what RDKit canonicalizes, c1ccncc1. Compared as strings they never match, and the query answered confidently with nothing.

That is fixed in its own commit (75307eb) and is independent of the NL layer — any caller passing {"compound": ...} to an eq on a smiles path hit it. Resolved SMILES are canonicalized once per name per search; a name resolving to something RDKit cannot parse is left alone rather than swallowed, since that is the resolver's fault to report.

Changes

• ord_schema/search/nl.py — ask, translate, answer, summarize, the client, and the error taxonomy (ModelRateLimitedError, ModelUnavailableError, MalformedQueryError, all under NLQueryError, which is what ord-interface maps onto status codes).
• ord_schema/search/nl_prompt.md — the system prompt as prose, including the rules that head off the failures measured on the cheap model (identifiers[*], quantifier scoping, reductions).
• pyproject.toml — an nl extra; ord-schema[search] still installs without anthropic, and dependencies_test.py now has a profile that proves it. The prompt ships as package data, and the tests extra pulls nl so a fresh install can collect the new tests.
• ord_schema/search/README.md — a usage section saying plainly what is and is not guaranteed.

Testing

uv run pytest -n auto — 1234 passed. Nothing in the suite reaches the network: a stub client returns real ToolUseBlock/TextBlock objects, which pins the coercion, the repair firing exactly once, the error mapping, the cacheable prefix, the forced tool call, and the summary staying bounded as the table grows.

End to end against the full corpus (2,428,291 reactions):

Q: which reactions use pyridine as a solvent?
   rows: 24930 in 5.0s
   text: The query returned 24,930 reaction records that used pyridine as a solvent...

Q: how many reactions were run above 350 K?
   rows: 1 in 2.2s
   text: 211,457 reactions were run above 350 K.

The canonicalization test fails without its fix and passes with it — checked by reverting the one line.

Notes

Stacked on #979, which adds the reduction this layer needs for "the highest-yielding reactions"; retarget to main once that lands.

The eval harness (task 7) is not here. It scores on which reactions come back rather than on query shape, and its cases need real reaction IDs from the corpus — that is the next PR.

🤖 Generated with Claude Code

Greptile Summary

Adds an optional natural-language search layer that translates English questions into validated ORD queries, executes them, and summarizes their results.

• Adds Anthropic-backed translation, one-turn query repair, bounded result summaries, and model error mapping.
• Canonicalizes resolver-produced SMILES before equality comparisons.
• Adds packaging metadata, documentation, dependency-profile coverage, and focused tests for the new functionality.

Confidence Score: 5/5

The PR appears safe to merge because no concrete changed-code failure or reachable security issue remains.

The new natural-language flow validates and compiles generated queries before execution, limits repair to one attempt, bounds summarized table data, and preserves the executed query, while the resolver canonicalization aligns name-resolved SMILES with corpus storage.

Important Files Changed

Filename	Overview
ord_schema/search/nl.py	Introduces the natural-language translation, validation/repair, execution orchestration, bounded summarization, and model error taxonomy; no actionable defect was established.
ord_schema/search/execute.py	Canonicalizes resolver-produced SMILES once per resolved name before string equality matching, preserving unparseable resolver output.
pyproject.toml	Adds the optional Anthropic-backed nl extra, includes it in test installations, and packages the Markdown prompt.
uv.lock	Locks Anthropic and its transitive dependencies; reported Keras and setuptools advisories concern unchanged pre-existing entries.
ord_schema/search/nl_test.py	Covers coercion, forced tool use, one-turn repair, error mapping, bounded summaries, query retention, and search timeout forwarding.

Sequence Diagram

sequenceDiagram
    participant U as Caller
    participant NL as search.nl
    participant M as Anthropic model
    participant C as Query compiler
    participant Corpus as Corpus
    U->>NL: ask(question, corpus)
    NL->>M: translate question with schema/tool
    M-->>NL: build_query tool input
    NL->>C: validate and compile
    alt invalid query
        C-->>NL: compiler error and suggestion
        NL->>M: one repair request
        M-->>NL: repaired tool input
        NL->>C: validate and compile again
    end
    NL->>Corpus: search(query, timeout)
    Corpus-->>NL: Arrow table
    NL->>M: question plus bounded summary
    M-->>NL: plain-text answer
    NL-->>U: Answer(query, table, text)

Loading

_{Reviews (1): Last reviewed commit: "Answer a question in English against a c..." | Re-trigger Greptile}