diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md new file mode 100644 index 0000000..4c7901d --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md @@ -0,0 +1,231 @@ +# What constrains a natural-language layer over the search grammar + +- **Date:** 2026-08-17 +- **Author:** Steven Kearnes +- **Status:** final (the design that follows from it is [beside this entry](assets/nl-search-design.md)) +- **Tags:** ord-schema, agents, search, natural-language, structured-outputs, anthropic, cost +- **License:** [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/) + +## Question + +`ord_schema.search` gives a model a validated `Query` grammar to write against instead of +SQL. The layer that turns a chemist's sentence into one of those queries has never +shipped: it exists on the unmerged `nl-query-backend` branch, written against a flat IR +that predates the grammar, and the `/ask` deployment it served has since been moved out +from under it. + +Rebuilding it raises one question that decides the whole design. Can the model be +*constrained* to emit a valid `Query` — structured outputs or a strict tool, where the +decoder itself refuses anything off-grammar — or does the layer have to accept whatever +comes back and check it afterwards? And with that settled: how good is a cheap model, and +what does a query cost? + +## Summary + +**Constrained decoding cannot carry this grammar.** Four separate walls, each hit +directly against the API rather than inferred: circular references are rejected outright, +the compiled grammar has a size budget, `oneOf` is unsupported, and there is a cap on +union-typed parameters. Stratifying the grammar into acyclic levels, requiring every +property, stripping descriptions and numeric constraints, and flattening the predicate +union each moved a wall — and the best that fits is a **depth-2** predicate tree, which +cannot express the nested correlation the pivots exist to serve. + +So translation is unconstrained generation, validated afterwards. That is fine: over ten +questions, **Opus 5 compiled 9/10 first try**, and **Haiku 4.5 reached 7/10 with one +repair turn** at **5.6× lower cost**. Both models get the shape wrong in the same +harmless way — they return the predicate tree as a JSON *string* — which a coercing parse +fixes deterministically. + +The probing also turned up a **gap in the grammar itself**: an aggregate or ordering key +cannot reach a value under a repeated level, so *"the ten highest-yielding reactions"* is +unwritable. Both models tried it, and repair does not help, because nothing they could +have written would have worked. + +## Method + +Every claim here is a live call against the Messages API, using the key the interface +deployment already holds (`ord-interface-anthropic-api-key`). The probes are in +[`assets/`](assets/README.md); the schema transforms they use — stratification and the +all-required rewrite — are the interesting half and are worth reading before repeating +this work. + +Ten questions were used throughout, from single-predicate (`reactions run above 350 K`) +to correlated (`pyridine as the solvent with a yield above 50%`) to aggregate (`the ten +highest-yielding Suzuki couplings`). A translation is scored on whether it validates +against the pydantic models and then compiles, which is a stricter bar than "looks right" +and a weaker one than "returns the right reactions" — result-level scoring wants an eval +set, which the design proposes and this entry does not have. + +## Findings + +### 1. The recursive grammar is refused by every constrained path + +`Query`'s predicate tree is recursive: `and`/`or` hold clauses, `not` holds a clause, and +a quantifier's body may hold another quantifier. Pydantic renders that as 12 definitions +with 134 internal references, 2,431 tokens. + +| path | result | +| --- | --- | +| `output_config.format` (structured outputs) | `Circular reference detected in schema definitions: And -> And` | +| tool with `strict: true` | same refusal | +| tool without `strict` | **accepted** | + +The SDK is not the obstacle: it sends the recursion faithfully, and its transform has +deliberate handling for `$defs` and root-level `$ref`. The refusal is server-side, and +the [structured outputs documentation](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) +confirms recursive schemas are unsupported. + +### 2. Removing the recursion does not help, because the wall behind it is smaller + +The grammar can be made acyclic without changing the IR: stratify it, so a level-*k* +predicate's clauses are level-*(k-1)* predicates and level 0 holds only leaves. Nothing is +inlined, so size grows by about **1,400 tokens per level** rather than by a power of four +— naive inlining reaches 331,086 tokens at depth 5, stratification 9,274. + +That clears the cycle and immediately hits the next wall: + +| depth | schema | `output_config.format` | strict tool | +| --- | --- | --- | --- | +| 0 | 2,425 tok | accepted | too large | +| 1 | 3,944 tok | too large | too large | +| 2 | 5,415 tok | too large | too large | +| 4 | 8,359 tok | too large | too large | + +Depth 0 is a single leaf predicate: no boolean, no quantifier. The strict-tool path is +tighter still and fits nothing. + +### 3. The budget is union-typed parameters, not tokens + +A [reported limit](https://github.com/anthropics/anthropic-sdk-python/issues/1185) says +optional properties roughly double the compiled state machine. Requiring every property +and expressing optionality as an explicit null takes the grammar from 13–17 optional +properties to zero and shrinks it by a third — and then two more walls appear in turn: +`oneOf` is unsupported (pydantic emits it for the discriminated union; rewriting to +`anyOf` is safe, since the models validate afterwards), and numeric constraints like +`exclusiveMinimum` are rejected. + +Past those, depth 0 fits at 1,000 tokens and depth 1 does not, at 1,653. The error +changes to *"too many parameters with union type"*, which names the real budget: every +nullable field is a union, and each level multiplies them. + +### 4. What does fit is a flat predicate at depth 2 — with every path enumerated + +Collapsing the eight predicate variants into one object whose `op` selects the shape, and +making `path` an enum, buys the most room. The enum turns out to be nearly free: + +| paths enumerated | depth 2 | depth 3 | depth 4 | +| --- | --- | --- | --- | +| free-form string | accepted | refused | refused | +| 40 | accepted | refused | refused | +| 120 | accepted | refused | refused | +| **all 431 scalar paths** | **accepted** (15,240 tok) | refused | refused | + +The enumeration is the 431 scalar paths `probe_flat_union.py` walks to, which skips the +key side of a map; the schema rendering counts 442 leaves and 537 lines because it carries +those too. + +This is worth remembering rather than building. Enumerating the paths makes an invented +column *structurally impossible*, and invented columns are the cheap model's main failure +mode. But depth 2 cannot nest a quantifier inside a quantifier, so correlated questions — +the ones the pivoted element index exists to answer — fall outside it. + +### 5. Unconstrained, both models write the tree as a string + +Given a non-strict forced tool call, the model reliably returns: + +```json +{"where": "{\"op\": \"and\", \"clauses\": [ ... ]}"} +``` + +The predicate tree arrives JSON-encoded inside a string, and pydantic rejects it. Opus did +this on 4 of 5 questions, Haiku on 5 of 5. Parsing string values back to objects before +validating takes both to 5/5 — this is the one thing a strict schema would have bought, +and it costs about ten lines to do without one. + +### 6. Opus 9/10, Haiku 7/10 with repair, at 5.6× the cost + +Ten questions, scored on compiling; a failure is handed back once with the compiler's own +error, which names the offending path and suggests a real one. + +| model | first try | after one repair | ~cost per query | +| --- | --- | --- | --- | +| Claude Opus 5 | 9/10 | 9/10 | 1.3¢ | +| Claude Haiku 4.5 | 5/10 | 7/10 | **0.23¢** | + +Haiku's failures are inventions of syntax the grammar does not have — `identifiers[*].value`, +`inputs[].components` — rather than misunderstandings of the question. Repair recovers two +of them. That is the failure mode finding 4 would eliminate outright, which is why the +path enum is worth keeping in mind for shallow queries. + +Cost is dominated by the cached prefix: the schema rendering plus the grammar is **15,481 +tokens on Opus, 11,462 on Haiku**, read at a tenth of list price on every call after the +first. Caching mattered more than model choice. + +### 7. The grammar cannot order by a value under a repeated level + +Opus's single failure, unrepairable and shared with Haiku: + +```text +outcomes.products.measurements.percentage.value: max needs a scalar column, not a repeated level +``` + +`Measure` and `Order` require a scalar path, and a yield lives under `outcomes` → +`products` → `measurements`. "The ten highest-yielding reactions" needs a per-reaction +reduction of a repeated path — `max` over each reaction's measurements — and there is no +way to write one. Both models reached for it independently, which is the strongest signal +available that the gap is real rather than a prompting artifact. + +## Conclusions / next steps + +Translation is unconstrained generation checked afterwards, and the design that follows +from that is [beside this entry](assets/nl-search-design.md): one forced tool call over +the recursive grammar with the prefix cached, a coercing parse, one repair turn carrying +the compiler's error, then execution and a short second call that writes the prose. + +Three things to do in order: + +1. **Close the reduction gap** (finding 7). It is a grammar change, it blocks a question + anyone would ask, and the NL layer would otherwise paper over it with a wrong answer. +2. **Build the layer**, with the model as configuration rather than a decision — the + measured 9/10 and 7/10 are ten questions, not an eval set. +3. **Then choose the model on evidence**, including whether path guidance in the cached + prefix closes Haiku's gap. At 5.6× it is worth the measurement. + +## What implementing it changed + +Three things the design got wrong, found by building it (ord-schema +[#979](https://github.com/open-reaction-database/ord-schema/pull/979), +[#980](https://github.com/open-reaction-database/ord-schema/pull/980), +[#981](https://github.com/open-reaction-database/ord-schema/pull/981)): + +**A resolved compound was compared in the wrong spelling.** The default resolver asks +PubChem, which returns pyridine as `C1=CC=NC=C1`; the projection stores what RDKit +canonicalizes, `c1ccncc1`. Compared as strings they never match, so the first real +question -- pyridine as a solvent -- answered **zero** over a corpus holding 24,930. +Wrong, and wrong in the way that looks like an answer. This predates the NL layer: any +caller passing `{"compound": ...}` to an `eq` on a smiles path hit it. + +**Forcing the tool call left no way to decline.** With `tool_choice` naming +`build_query`, a model asked for something the grammar cannot express -- comparing two +columns -- builds a plausible query that means something else. Both models did. A +`cannot_answer` tool beside it, with `tool_choice: any`, turns that into a refusal +carrying the model's own reason. + +**An eval case built by sampling failed correct translations.** The counterexamples for +"a desired product with a yield above 50%" were derived by differencing a 200-row sample +of the right query against a 400-row sample of the wrong one -- set arithmetic on +samples. Both models failed a case they had answered correctly, and a round of prompt +tuning went into a problem that did not exist. Asking for the counterexamples directly +-- `near_miss AND NOT reference` -- is exact, and with honest cases both models pass 5/5. + +The second and third are the same lesson from opposite directions: a layer that cannot +say "no" invents an answer, and a test that cannot say "wrong" invents a failure. + +## References + +- [`assets/nl-search-design.md`](assets/nl-search-design.md) — the design this entry argues for +- [`assets/nl-search-plan.md`](assets/nl-search-plan.md) — the task-by-task plan built from it +- [Structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) — recursion unsupported; limits undocumented +- [anthropic-sdk-python#1185](https://github.com/anthropics/anthropic-sdk-python/issues/1185) — the optional-property state-space report +- [ord-schema#966](https://github.com/open-reaction-database/ord-schema/pull/966) — the split that renamed `agent` to `search` +- [Where the agent search cache can live](../2026-08-15-where-the-search-cache-lives/README.md) — the pivots and index this layer queries through diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md new file mode 100644 index 0000000..6bcdc9a --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md @@ -0,0 +1,50 @@ +# Scripts for "What constrains a natural-language layer over the search grammar" + +- **Date:** 2026-08-17 +- **Author:** Steven Kearnes +- **License:** [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/) + +Probes for [the entry beside them](../README.md), the design they argue for +([`nl-search-design.md`](nl-search-design.md)), and the plan that implements it +([`nl-search-plan.md`](nl-search-plan.md)). + +Every probe but the first makes live Messages API calls. Run them with `ord_schema` +importable and a key in the environment; the interface deployment's key works, and is +where these numbers came from: + +```bash +export ANTHROPIC_API_KEY=$(aws secretsmanager get-secret-value \ + --secret-id ord-interface-anthropic-api-key --query SecretString --output text) +uv run --with anthropic python probe_repair.py claude-haiku-4-5 +``` + +| script | what it measures | finding | +| --- | --- | --- | +| `probe_recursion.py` | what the SDK puts on the wire for a recursive model, via a mock transport — no key needed | 1 | +| `probe_recursion_live.py` | whether `output_config.format` accepts the recursive grammar | 1 | +| `probe_tool_recursion.py` | the same question for tool schemas, strict and not | 1, 5 | +| `probe_shapes.py` | how often a forced tool call parses as written, and after coercion | 5, 6 | +| `probe_stratified.py` | whether an acyclic grammar unlocks structured outputs | 2 | +| `probe_bisect.py` | which construct exhausts the compiled-grammar budget | 2, 3 | +| `probe_strict_tool.py` | whether the strict-tool path has a larger budget | 2 | +| `probe_required.py` | whether requiring every property fits the budget | 3 | +| `probe_flat_union.py` | a flattened predicate with paths enumerated, by depth | 4 | +| `probe_repair.py` | first-try and after-repair accuracy, per model, with cost | 6, 7 | + +Two of these are library rather than probe, and are the reusable part: + +- `stratify.py` turns a recursive JSON Schema into an acyclic one by numbering levels — + a level-*k* definition references level *(k-1)*, and level 0 drops the recursive + branches. Nothing is inlined, so size grows per level rather than per power. +- `require_all.py` makes every property required with an explicit null instead of + optional, rewrites `oneOf` to `anyOf`, and drops keywords a decoder rejects. Optional + properties are what multiply the compiled state machine. + +Both transform only the schema *shown to the model*. The pydantic models are untouched and +still do the validating, which is why dropping `exclusiveMinimum` and the discriminator +costs no correctness. + +`probe_repair.py` takes a model id as its argument and is the one to re-run when the +prompt changes; it prints per-question outcomes, first-try and after-repair tallies, and +the token counts the cost figures come from. Costs in the entry are computed from those +counts at list price, with cached reads at a tenth. diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-design.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-design.md new file mode 100644 index 0000000..4376a1e --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-design.md @@ -0,0 +1,182 @@ +# Natural-language search over the ORD corpus — design + +- **Date:** 2026-08-17 +- **Author:** Steven Kearnes +- **Status:** draft, for review +- **License:** [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/) + +The measurements this argues from are in [the entry beside it](../README.md). Read finding +1–4 before proposing constrained decoding again; it was tried four ways and does not fit. + +## What this builds + +A question in English becomes a validated `Query`, runs against a `Corpus`, and comes back +with the reactions and a sentence saying what they show. One entry point: + +```python +from ord_schema.search import execute, nl + +corpus = execute.Corpus("projections/*/*.parquet", "structures/*/*.parquet") +answer = nl.ask("which reactions use pyridine as a solvent and beat 50% yield?", corpus) + +answer.query # the Query that ran, for display and for a "run it again" button +answer.table # pyarrow.Table, exactly what corpus.search returned +answer.text # "1,204 reactions. Most are Suzuki couplings; yields cluster at 60-75%." +``` + +`ask` is the whole public surface. `translate` and `answer` are separable because the eval +harness needs the first without the second, and a caller that renders its own results +needs the first without the third. + +## What it does not build + +- **Multi-turn.** No clarifying questions, no follow-ups, no session state. A question is + answered or it fails. +- **A second backend.** Corpus only. ord-interface's Postgres path is out of scope, and + the broken `ord_schema.agent.nl_query` import there is a separate cleanup. +- **Constrained decoding.** Measured dead; see the entry. +- **Text search tuning.** Substring predicates remain expressible and remain unoptimized. + +## The gap this depends on + +`Measure` and `Order` require a scalar path, so an aggregate cannot reach a value under a +repeated level: + +```text +outcomes.products.measurements.percentage.value: max needs a scalar column, not a repeated level +``` + +"The ten highest-yielding reactions" is therefore unwritable, and both models tried to +write it anyway. **Task 1 closes this before any NL code exists**, because a layer built +over the gap would answer the question wrongly rather than refuse it. + +The shape: a reduction over a repeated path, usable wherever a scalar is wanted. + +```json +{"order_by": [{"key": {"reduce": "max", "path": "outcomes.products.measurements.percentage.value"}, "descending": true}], + "limit": 10} +``` + +It compiles to a list aggregate over the reaction's own elements — `list_max(list_transform(...))` +over the projection, or `max(...)` over the level's pivot with a correlated subquery — so +it inherits the routing the executor already does per quantifier. `min`, `max`, `avg`, +`sum`, and `count` all make sense; `count` needs no path. + +Filtering on a reduction (`reactions whose best yield beats 50%`) is already expressible as +`exists ... where percentage.value > 50`, and stays that way. This is only about ordering +and aggregating, where there is no quantifier to hang the condition on. + +## How translation works + +```mermaid +flowchart TB + Q["question"] --> T["forced tool call
build_query, non-strict"] + T --> C["coerce: JSON strings to objects"] + C --> V["Query.model_validate"] + V --> K["compile_query"] + K -- "ValidationError or QueryError" --> R{"repaired once?"} + R -- no --> T2["hand back the error
+ its did-you-mean"] + T2 --> C + R -- yes --> F["MalformedQueryError"] + K -- ok --> X["corpus.search"] + X --> S["summarize: counts, columns, a few rows"] + S --> A["second call writes the prose"] +``` + +**The cached prefix** is the system text (instructions plus `schema.describe()`) and the +tool definition, marked `cache_control: {"type": "ephemeral", "ttl": "1h"}`. Measured at +15,481 tokens on Opus and 11,462 on Haiku, read at a tenth of list price on every call +after the first. Two rules keep it cached: nothing volatile goes in the prefix (no +timestamps, no per-request identifiers), and the tool list is built once at import. + +**The coercing parse** walks the tool input and replaces any string that parses as JSON +with the parsed value, because both models return the predicate tree as a JSON string +(finding 5). It runs before pydantic, and it is not a fallback — it is the normal path. + +**One repair turn.** On a validation or compile failure, the assistant's tool call and a +`tool_result` carrying the error text go back with `is_error: true`. The compiler's errors +already name the bad path and suggest a real one, which is what makes this worth doing; +Haiku recovers two of five failures on it. A second failure raises rather than looping — +an unbounded repair loop spends real money discovering the model cannot answer. + +**The answer call** sees a summary, never the table: row count, column names, and up to +five sample rows rendered as text. A result set of 100,000 reactions costs the same +prompt as one of three. + +## Module layout + +| file | holds | +| --- | --- | +| `ord_schema/search/nl.py` | `ask`, `translate`, `answer`, the client, the errors | +| `ord_schema/search/nl_prompt.md` | the system prompt, as markdown rather than a Python string | +| `ord_schema/search/nl_eval.py` | `EvalCase`, `load_cases`, scoring, the report | +| `ord_schema/search/nl_cases.yaml` | the eval set, versioned beside the prompt it grades | + +`nl.py` sits in `search/` because it depends on the grammar, the compiler, and the +executor, and nothing else depends on it. The prompt lives beside it for the reason the +old branch found: a prompt in a markdown file can be edited and diffed as prose. + +Dependencies: a new `nl` extra carrying `anthropic`, since `ord-schema[search]` must stay +installable without it. `ord_schema/dependencies_test.py` gains a profile row, which is +what keeps that promise honest. + +## Errors + +The names come from the `nl-query-backend` branch, because ord-interface already maps them +to status codes and there is no reason to invent new ones: + +| error | when | +| --- | --- | +| `ModelRateLimitedError` | 429 from the API | +| `ModelUnavailableError` | 5xx, timeout, connection failure | +| `MalformedQueryError` | still invalid after the repair turn | + +All three inherit `NLQueryError`. Everything else propagates: a `PairingError` from the +corpus is not the model's fault and should not be dressed up as one. + +## Testing + +**Unit tests take no network.** A fake client returns canned tool calls, which is enough +to pin every behavior that matters: the coercion, the repair turn firing exactly once, the +error taxonomy, the prefix being marked cacheable, and the summary staying bounded as the +table grows. + +**The eval harness is opt-in and costs money.** `nl_eval.py` runs cases against a real +model and reports first-try and after-repair rates, per-question failures, and token cost. +A case states a question and what the answer must satisfy — not a `Query` literal, since +several spellings are right: + +```yaml +- question: reactions using pyridine as the solvent + compiles: true + must_return: [ord-1f2a..., ord-9c7e...] # reactions any correct query returns + must_not_return: [ord-4b81...] # a near-miss the wrong quantifier would include +``` + +Scoring on returned reactions rather than on query shape is what makes the harness able to +say a translation is *wrong* rather than merely *different*. The cases live beside the +prompt so that changing one and not the other shows up in review. + +## What "fast" means here + +A question costs one translation call, one execution, and one answer call. The middle term +is the only one this repository controls, and it is already measured: a structure query +lands in 0.02–1.5 s warm, a quantifier over a pivot in tens of milliseconds. Translation is +seconds and dominated by output tokens; the answer call is shorter. + +So the latency work is not in this layer, and the design should not pretend otherwise. What +this layer owes the corpus is *not making it slow*: no query without a limit, the timeout +passed through to `corpus.search`, and the model never handed enough rows to think it should +summarize them itself. + +## Open questions + +1. **Which model.** Ten questions say Opus 9/10 and Haiku 7/10 at 5.6× less. The eval set + decides, and the interesting variable is whether worked path examples in the cached + prefix close the gap — they are nearly free once cached. +2. **Whether the path enum earns a place.** Enumerating all 431 scalar paths fits a depth-2 + grammar and makes invented paths impossible. If the evals show shallow queries dominate + real traffic, a constrained fast path with an unconstrained fallback becomes tempting — + but it is two systems, and it needs the traffic to justify it. +3. **Where `/ask` lands.** Out of scope here, and the answer probably changes once the + layer exists and ord-interface's dependency on the deleted module is cleaned up. diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-plan.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-plan.md new file mode 100644 index 0000000..c457af3 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-plan.md @@ -0,0 +1,1073 @@ +# Natural-language search implementation plan + +- **Date:** 2026-08-17 +- **Author:** Steven Kearnes +- **Status:** draft +- **License:** [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A chemist's question becomes a validated `Query`, runs against a `Corpus`, and comes back with the reactions and a sentence about them. + +**Architecture:** One forced tool call over the recursive grammar with the ~15K-token prefix cached; the response is coerced (both models return the predicate tree as a JSON string), validated by the existing pydantic models, and compiled. A failure is handed back once with the compiler's own error, which names the bad path and suggests a real one. Execution goes through `Corpus.search`; a second short call sees a summary of the result — never the table — and writes the prose. + +**Tech Stack:** Python 3.11+, pydantic 2, `anthropic` SDK, DuckDB via `ord_schema.search.execute`, pytest. + +**Spec:** [`nl-search-design.md`](nl-search-design.md), and the measurements it argues from in [the entry](../README.md). + +## Global Constraints + +- **Model is configuration, never a hard-coded decision.** `DEFAULT_MODEL = "claude-haiku-4-5"`; every entry point takes `model`. +- **The cached prefix must stay byte-stable.** No timestamps, no per-request identifiers, no dict iteration order in the system text or the tool definition. Cache reads are ~90% of the per-query cost. +- **`ord-schema[search]` must remain installable without `anthropic`.** The new dependency goes in an `nl` extra, and `ord_schema/dependencies_test.py` gains a profile row that proves it. +- **Repair runs at most once.** A second failure raises `MalformedQueryError`. +- **No lazy imports.** Every import at module top, per the repository's Python style. +- **Docstrings are Google style**, summary line on one physical line, `Args:`/`Returns:`/`Raises:` where non-empty. +- **Ruff formats at line length 88**; `uv run ruff format` and `uv run ruff check` must pass before every commit. +- **Tests run offline.** Only `nl_eval.py` calls the API, and nothing in the suite invokes it. + +--- + +## File structure + +| file | responsibility | +| --- | --- | +| `ord_schema/search/query.py` (modify) | `Reduction`, and the reduction arms of `Order.key` and `Measure.path` | +| `ord_schema/search/nl.py` (create) | errors, client, `translate`, `summarize`, `answer`, `ask` | +| `ord_schema/search/nl_prompt.md` (create) | the system prompt, as prose | +| `ord_schema/search/nl_test.py` (create) | every behavior above, against a stub client | +| `ord_schema/search/nl_eval.py` (create) | `EvalCase`, `load_cases`, `run_case`, `report` | +| `ord_schema/search/nl_cases.yaml` (create) | the eval set | +| `pyproject.toml` (modify) | the `nl` extra | +| `ord_schema/dependencies_test.py` (modify) | the `nl` profile | + +--- + +### Task 1: A reduction over a repeated path + +The grammar cannot order by a value under a repeated level, so "the ten highest-yielding reactions" is unwritable. `resolve()` already returns a *list* expression for a repeated path, so a reduction is one DuckDB list aggregate around it. + +**Files:** + +- Modify: `ord_schema/search/query.py` (`Order`, `Measure`, `compile_query`) +- Test: `ord_schema/search/query_test.py` + +**Interfaces:** + +- Consumes: `resolve(path, schema=...) -> _Resolved(expression, repeated, dtype)`, `QueryError`. +- Produces: `Reduction` with fields `reduce: Literal["min","max","avg","sum","count"]` and `path: str`; `Order.key: str | Reduction`; `Measure.path: str | Reduction | None`. + +- [ ] **Step 1: Write the failing test** + +```python +def test_ordering_by_a_reduction_over_a_repeated_path(): + compiled = query.compile_query( + query.Query.model_validate( + { + "order_by": [ + { + "key": { + "reduce": "max", + "path": "outcomes.products.measurements.percentage.value", + }, + "descending": True, + } + ], + "limit": 10, + } + ) + ) + assert "list_max(" in compiled.sql + assert compiled.sql.endswith("DESC LIMIT 10") + + +def test_a_reduction_over_a_scalar_path_is_refused(): + # A scalar needs no reducing, and accepting one would give two spellings for the + # same query -- one of which silently wraps a value in a single-element list. + with pytest.raises(query.QueryError, match="already scalar"): + query.compile_query( + query.Query.model_validate( + { + "order_by": [ + { + "key": { + "reduce": "max", + "path": "conditions.temperature.setpoint_kelvin", + } + } + ] + } + ) + ) +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest ord_schema/search/query_test.py -k reduction -v` +Expected: FAIL — `Order.key` rejects a dict, so pydantic raises a `ValidationError`. + +- [ ] **Step 3: Add the model and the compiler support** + +In `ord_schema/search/query.py`, above `class Order`: + +```python +# DuckDB's list aggregates, which ignore nulls; count is the number of values that are +# actually there, so it filters rather than taking len() of a list holding nulls. +_REDUCERS = { + "min": "list_min({expression})", + "max": "list_max({expression})", + "avg": "list_avg({expression})", + "sum": "list_sum({expression})", + "count": "len(list_filter({expression}, value -> value IS NOT NULL))", +} + + +class Reduction(BaseModel): + """One value per reaction, reduced from a path that crosses a repeated level. + + An ordering key and an aggregate's argument both have to be scalar, which leaves + "the highest-yielding reactions" unwritable: a yield lives under outcomes, + products, and measurements, so the path resolves to a list. This reduces that list + to the one number the reaction is judged by. + + Attributes: + reduce: How to reduce the list to a value. + path: A dotted path that crosses at least one repeated level. + """ + + reduce: Literal["min", "max", "avg", "sum", "count"] + path: str +``` + +And the resolver, beside `_scalar`: + +```python +def _reduced(reduction: Reduction, schema: pa.Schema) -> str: + """Returns the expression reducing a repeated path to one value per reaction. + + Args: + reduction: What to reduce, and how. + schema: Schema the path is resolved against. + + Returns: + A DuckDB expression yielding one scalar per reaction, NULL where the reaction + holds no elements at all. + + Raises: + QueryError: If the path does not cross a repeated level, since a scalar path + needs no reduction and accepting one would give a query two spellings. + """ + resolved = resolve(reduction.path, schema=schema) + if not resolved.repeated: + raise QueryError( + f"{reduction.path}: {reduction.reduce} reduces a repeated level, and this " + f"path is already scalar; order by the path itself" + ) + return _REDUCERS[reduction.reduce].format(expression=resolved.expression) +``` + +Widen the two fields: + +```python +class Order(BaseModel): + """How to sort the result.""" + + key: str | Reduction + descending: bool = False +``` + +```python + fn: Literal["count", "count_distinct", "sum", "avg", "min", "max"] + path: str | Reduction | None = None + name: str +``` + +- [ ] **Step 4: Teach `compile_query` both places** + +The measure argument, replacing `argument = _scalar(measure.path, schema, measure.fn)`: + +```python + elif isinstance(measure.path, Reduction): + argument = _reduced(measure.path, schema) + else: + argument = _scalar(measure.path, schema, measure.fn) +``` + +The ordering key, at the top of the `for order in query.order_by:` body: + +```python + if isinstance(order.key, Reduction): + key = _reduced(order.key, schema) + elif orderable is None: +``` + +An aggregated query still orders by a measure name or a `group_by` path; a reduction is a per-reaction value and there is no such row after grouping. Leave that arm as it is — a `Reduction` reaching it is caught by the `isinstance` above only for the ungrouped case, so add the guard: + +```python + if isinstance(order.key, Reduction) and orderable is not None: + raise QueryError( + "an aggregated query orders by a measure name or a group_by path; " + "reduce inside a measure instead" + ) +``` + +- [ ] **Step 5: Run the tests** + +Run: `uv run pytest ord_schema/search/query_test.py -v` +Expected: PASS, including the two new tests. + +- [ ] **Step 6: Check the reduction against real data** + +Run: + +```bash +uv run python -c " +from ord_schema.search import execute, query +corpus = execute.Corpus('~/ord/projections/**/*.parquet', '~/ord/structures/**/*.parquet', + require_current=False, resolver={}.__getitem__) +table = corpus.search(query.Query.model_validate({ + 'order_by': [{'key': {'reduce': 'max', 'path': 'outcomes.products.measurements.percentage.value'}, + 'descending': True}], + 'limit': 10})) +print(table.num_rows) +" +``` + +Expected: 10 rows. This is the question both models tried to write and could not. + +- [ ] **Step 7: Document it** + +Add a row to the worked-example table in `ord_schema/search/README.md`: + +```markdown +| the ten highest-yielding reactions | `order_by` a `reduce` over `outcomes.products.measurements` | no quantifier: a list aggregate over the projection | +``` + +- [ ] **Step 8: Commit** + +```bash +git add ord_schema/search/query.py ord_schema/search/query_test.py ord_schema/search/README.md +git commit -m "Let a query order by a value under a repeated level" +``` + +--- + +### Task 2: The package, its dependency, and its errors + +**Files:** + +- Create: `ord_schema/search/nl.py`, `ord_schema/search/nl_test.py` +- Modify: `pyproject.toml`, `ord_schema/dependencies_test.py` + +**Interfaces:** + +- Produces: `NLQueryError`, `ModelUnavailableError`, `ModelRateLimitedError`, `MalformedQueryError`, `DEFAULT_MODEL = "claude-haiku-4-5"`, `get_client() -> anthropic.Anthropic`. + +- [ ] **Step 1: Write the failing test** + +```python +def test_the_errors_share_one_base(): + # ord-interface maps these onto status codes, so a caller can catch the base and + # still tell a rate limit from a query it could not build. + assert issubclass(nl.ModelRateLimitedError, nl.NLQueryError) + assert issubclass(nl.ModelUnavailableError, nl.NLQueryError) + assert issubclass(nl.MalformedQueryError, nl.NLQueryError) +``` + +- [ ] **Step 2: Run it** + +Run: `uv run pytest ord_schema/search/nl_test.py -v` +Expected: FAIL — `ModuleNotFoundError: ord_schema.search.nl`. + +- [ ] **Step 3: Write the module head** + +```python +"""Natural-language questions, translated into the search grammar and answered. + +A model cannot be constrained to emit a valid Query: the grammar is recursive, and both +constrained-decoding paths refuse it -- the measurements are in the ord-logbook entry +"What constrains a natural-language layer over the search grammar". So translation is +generation checked afterwards: the response is coerced, validated by the same models the +compiler uses, and handed back once with the compiler's error when it does not compile. +""" + +import dataclasses +import json +from importlib import resources +from typing import Any + +import anthropic +import pyarrow as pa + +from ord_schema.logging import get_logger +from ord_schema.search import execute, query, schema + +logger = get_logger(__name__) + +DEFAULT_MODEL = "claude-haiku-4-5" +MAX_TOKENS = 2048 + + +class NLQueryError(Exception): + """A question could not be answered.""" + + +class ModelUnavailableError(NLQueryError): + """The model could not be reached.""" + + +class ModelRateLimitedError(NLQueryError): + """The model refused the request for rate reasons.""" + + +class MalformedQueryError(NLQueryError): + """The model's query did not validate, and the repair attempt did not either.""" + + +def get_client() -> anthropic.Anthropic: + """Returns a client reading its credentials from the environment.""" + return anthropic.Anthropic() +``` + +- [ ] **Step 4: Run it** + +Run: `uv run pytest ord_schema/search/nl_test.py -v` +Expected: PASS. + +- [ ] **Step 5: Declare the dependency** + +In `pyproject.toml`, after the `search` extra: + +```toml +# Optional: ord_schema.search.nl turns a question into a Query by asking a model, so it +# needs an API client where the rest of the search subpackage needs none. +nl = [ + "anthropic>=0.120", + "ord-schema[search]", +] +``` + +In `ord_schema/dependencies_test.py`, add to both dictionaries: + +```python + "nl": ("ord_schema.search.nl",), +``` + +```python + "nl": ("search", "nl"), +``` + +- [ ] **Step 6: Run the dependency test** + +Run: `uv run pytest ord_schema/dependencies_test.py -v` +Expected: PASS — five profiles, none skipped, since `anthropic` is installed in a development checkout. + +- [ ] **Step 7: Commit** + +```bash +git add ord_schema/search/nl.py ord_schema/search/nl_test.py pyproject.toml ord_schema/dependencies_test.py +git commit -m "Add the natural-language module, its extra, and its errors" +``` + +--- + +### Task 3: Translation, with the coercing parse + +**Files:** + +- Create: `ord_schema/search/nl_prompt.md` +- Modify: `ord_schema/search/nl.py`, `ord_schema/search/nl_test.py` + +**Interfaces:** + +- Consumes: `schema.describe()`, `query.Query`, `query.compile_query`, the errors from Task 2. +- Produces: `translate(question, *, client=None, model=DEFAULT_MODEL, repair=True) -> query.Query`; `SYSTEM_PROMPT: str`; `TOOL: dict`. + +- [ ] **Step 1: Write the failing tests** + +```python +class _StubClient: + """Returns canned tool calls, and records what it was asked.""" + + def __init__(self, *inputs): + self._inputs = list(inputs) + self.requests = [] + self.messages = self + + def create(self, **kwargs): + self.requests.append(kwargs) + payload = self._inputs.pop(0) + block = types.SimpleNamespace( + type="tool_use", id="toolu_stub", name="build_query", input=payload + ) + usage = types.SimpleNamespace( + input_tokens=1, output_tokens=1, + cache_creation_input_tokens=0, cache_read_input_tokens=0, + ) + return types.SimpleNamespace(content=[block], usage=usage, stop_reason="tool_use") + + +_WHERE = { + "op": "exists", + "path": "inputs.components", + "where": {"op": "eq", "path": "reaction_role", "value": {"literal": "SOLVENT"}}, +} + + +def test_a_tree_returned_as_a_json_string_is_still_understood(): + # Both models do this, most of the time: the predicate arrives JSON-encoded inside + # a string rather than as an object. + client = _StubClient({"where": json.dumps(_WHERE)}) + result = nl.translate("solvent reactions", client=client) + assert result.where.op == "exists" + + +def test_the_prefix_is_marked_cacheable(): + # Cache reads are most of what a query costs; an uncached prefix is a 10x bill. + client = _StubClient({"where": _WHERE}) + nl.translate("solvent reactions", client=client) + system = client.requests[0]["system"] + assert system[0]["cache_control"] == {"type": "ephemeral", "ttl": "1h"} + + +def test_the_schema_reaches_the_prompt(): + client = _StubClient({"where": _WHERE}) + nl.translate("solvent reactions", client=client) + assert "reaction_role" in client.requests[0]["system"][0]["text"] +``` + +- [ ] **Step 2: Run them** + +Run: `uv run pytest ord_schema/search/nl_test.py -v` +Expected: FAIL — `AttributeError: module 'ord_schema.search.nl' has no attribute 'translate'`. + +- [ ] **Step 3: Write the prompt** + +`ord_schema/search/nl_prompt.md`: + +```markdown +You turn a chemist's question into one ORD search query by calling `build_query`. + +Rules that keep a query answerable: + +- Paths are dotted names from the schema below. There is no array syntax: write + `inputs.components`, never `inputs[].components` or `identifiers[*].value`. +- Any path that crosses a repeated level must be bound by `exists` or `forall`, and the + paths inside that quantifier are relative to the bound element. +- Two conditions on the *same* element go inside one quantifier. Two conditions on + *different* elements are two quantifiers. +- Name compounds rather than spelling structures: `{"compound": "pyridine"}` resolves to + SMILES. Use `substructure` with a SMARTS only for a pattern the user describes. +- To rank by a value under a repeated level, order by a reduction: + `{"reduce": "max", "path": "outcomes.products.measurements.percentage.value"}`. +- Prefer the smallest query that answers the question, and set `limit` when the user + asks for a number of results. + +The corpus schema, as an indented type tree in DuckDB's types: +``` + +- [ ] **Step 4: Implement translation** + +```python +SYSTEM_PROMPT = ( + (resources.files("ord_schema.search") / "nl_prompt.md").read_text(encoding="utf-8") + + "\n\n" + + schema.describe() +) + +# Built once: the tool definition is part of the cached prefix, so a dict rebuilt per +# call with a different key order would silently cost a cache miss. +TOOL: dict[str, Any] = { + "name": "build_query", + "description": "Build an ORD search query from the user's question.", + "input_schema": query.Query.model_json_schema(), +} +_SYSTEM = [ + { + "type": "text", + "text": SYSTEM_PROMPT, + "cache_control": {"type": "ephemeral", "ttl": "1h"}, + } +] + + +def _coerce(value: Any) -> Any: + """Returns the value with JSON-encoded strings parsed back into objects. + + A model handed a recursive tool schema usually returns the nested predicate as a + JSON string rather than an object, which pydantic rejects. Parsing it back is the + normal path rather than a fallback. + """ + if isinstance(value, str): + try: + return _coerce(json.loads(value)) + except (json.JSONDecodeError, TypeError): + return value + if isinstance(value, dict): + return {key: _coerce(item) for key, item in value.items()} + if isinstance(value, list): + return [_coerce(item) for item in value] + return value + + +def _ask_model(client, model, messages): + """Returns the tool_use block from one forced call, mapping API failures.""" + try: + response = client.messages.create( + model=model, + max_tokens=MAX_TOKENS, + system=_SYSTEM, + messages=messages, + tools=[TOOL], + tool_choice={"type": "tool", "name": "build_query"}, + ) + except anthropic.RateLimitError as error: + raise ModelRateLimitedError(str(error)) from error + except (anthropic.APIConnectionError, anthropic.APIStatusError) as error: + raise ModelUnavailableError(str(error)) from error + for block in response.content: + if block.type == "tool_use": + return block + raise MalformedQueryError("the model returned no query") +``` + +- [ ] **Step 5: Add `translate` itself** + +```python +def translate( + question: str, + *, + client: anthropic.Anthropic | None = None, + model: str = DEFAULT_MODEL, + repair: bool = True, +) -> query.Query: + """Returns the query a question asks for. + + Args: + question: The question, in English. + client: Anthropic client; one is built from the environment if omitted. + model: Which model translates. Cheap models need the repair turn more often. + repair: Hand a failure back once with the compiler's error. Off for measuring + first-try accuracy. + + Returns: + A Query that compiles against the projection schema. + + Raises: + MalformedQueryError: If the query does not validate or compile, after the + repair turn if one was allowed. + ModelRateLimitedError: If the model is rate limited. + ModelUnavailableError: If the model cannot be reached. + """ + client = client if client is not None else get_client() + messages: list[dict[str, Any]] = [{"role": "user", "content": question}] + block = _ask_model(client, model, messages) + try: + return _validated(block.input) + except (ValueError, query.QueryError) as error: + if not repair: + raise MalformedQueryError(str(error)) from error + first = error + logger.info("repairing a query that did not compile: %s", first) + messages += [ + {"role": "assistant", "content": [{"type": "tool_use", "id": block.id, + "name": block.name, "input": block.input}]}, + {"role": "user", "content": [{"type": "tool_result", "tool_use_id": block.id, + "is_error": True, + "content": f"That query was rejected: {first}. " + "Call build_query again with it fixed."}]}, + ] + retry = _ask_model(client, model, messages) + try: + return _validated(retry.input) + except (ValueError, query.QueryError) as error: + raise MalformedQueryError(str(error)) from error + + +def _validated(raw: Any) -> query.Query: + """Returns the Query a tool call carries, proven to compile.""" + parsed = query.Query.model_validate(_coerce(raw)) + query.compile_query(parsed) + return parsed +``` + +- [ ] **Step 6: Run the tests** + +Run: `uv run pytest ord_schema/search/nl_test.py -v` +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add ord_schema/search/nl.py ord_schema/search/nl_prompt.md ord_schema/search/nl_test.py +git commit -m "Translate a question into a query, coercing what the model returns" +``` + +--- + +### Task 4: The repair turn, pinned + +**Files:** + +- Modify: `ord_schema/search/nl_test.py` + +**Interfaces:** + +- Consumes: `translate`, `_StubClient` from Task 3. + +- [ ] **Step 1: Write the failing tests** + +```python +_BAD_PATH = {"op": "eq", "path": "identifiers[*].value", "value": {"literal": "x"}} + + +def test_a_bad_path_is_handed_back_once_and_recovered(): + client = _StubClient({"where": _BAD_PATH}, {"where": _WHERE}) + result = nl.translate("aspirin reactions", client=client) + assert result.where.op == "exists" + assert len(client.requests) == 2 + + +def test_the_repair_carries_the_compiler_s_suggestion(): + client = _StubClient({"where": _BAD_PATH}, {"where": _WHERE}) + nl.translate("aspirin reactions", client=client) + sent = client.requests[1]["messages"][-1]["content"][0]["content"] + assert "did you mean" in sent + + +def test_a_second_failure_raises_rather_than_looping(): + client = _StubClient({"where": _BAD_PATH}, {"where": _BAD_PATH}) + with pytest.raises(nl.MalformedQueryError, match="identifiers"): + nl.translate("aspirin reactions", client=client) + assert len(client.requests) == 2 + + +def test_repair_can_be_turned_off_for_measurement(): + client = _StubClient({"where": _BAD_PATH}) + with pytest.raises(nl.MalformedQueryError): + nl.translate("aspirin reactions", client=client, repair=False) + assert len(client.requests) == 1 +``` + +- [ ] **Step 2: Run them** + +Run: `uv run pytest ord_schema/search/nl_test.py -k repair -v` +Expected: PASS if Task 3 was implemented as written; a failure here means the repair path is wrong, not that the test is. + +- [ ] **Step 3: Commit** + +```bash +git add ord_schema/search/nl_test.py +git commit -m "Pin the repair turn to exactly one attempt" +``` + +--- + +### Task 5: The summary and the prose answer + +**Files:** + +- Modify: `ord_schema/search/nl.py`, `ord_schema/search/nl_test.py` + +**Interfaces:** + +- Produces: `summarize(table, *, rows=5) -> str`; `answer(question, table, *, client=None, model=DEFAULT_MODEL) -> str`. + +- [ ] **Step 1: Write the failing tests** + +```python +def test_the_summary_is_bounded_by_the_row_cap_not_the_table(): + small = pa.table({"reaction_id": ["a", "b"]}) + large = pa.table({"reaction_id": [str(i) for i in range(100_000)]}) + assert len(nl.summarize(large)) < 2 * len(nl.summarize(small)) + + +def test_the_summary_states_the_row_count(): + assert "100000 rows" in nl.summarize(pa.table({"reaction_id": [str(i) for i in range(100_000)]})) +``` + +- [ ] **Step 2: Run them** + +Run: `uv run pytest ord_schema/search/nl_test.py -k summary -v` +Expected: FAIL — `summarize` does not exist. + +- [ ] **Step 3: Implement both** + +```python +def summarize(table: pa.Table, *, rows: int = 5) -> str: + """Returns a description of a result small enough to put in a prompt. + + Args: + table: What the search returned. + rows: How many sample rows to show. + + Returns: + The row count, the column names, and up to ``rows`` rows. A result of a hundred + thousand reactions costs the same prompt as one of three. + """ + sample = table.slice(0, rows).to_pylist() + lines = [f"{table.num_rows} rows, columns: {', '.join(table.column_names)}"] + lines += [json.dumps(row, default=str) for row in sample] + if table.num_rows > rows: + lines.append(f"... {table.num_rows - rows} more rows not shown") + return "\n".join(lines) + + +def answer( + question: str, + table: pa.Table, + *, + client: anthropic.Anthropic | None = None, + model: str = DEFAULT_MODEL, +) -> str: + """Returns a sentence or two saying what the result shows. + + Args: + question: The question the result answers. + table: What the search returned. + client: Anthropic client; one is built from the environment if omitted. + model: Which model writes the prose. + + Returns: + Plain text, with no markdown and no invented chemistry: the model sees a summary + rather than the rows, so it can only describe what the summary states. + + Raises: + ModelRateLimitedError: If the model is rate limited. + ModelUnavailableError: If the model cannot be reached. + """ + client = client if client is not None else get_client() + try: + response = client.messages.create( + model=model, + max_tokens=512, + system=( + "You describe the result of a database query in one or two plain " + "sentences. State only what the summary shows. Do not invent " + "chemistry, and do not use markdown." + ), + messages=[ + { + "role": "user", + "content": f"Question: {question}\n\nResult:\n{summarize(table)}", + } + ], + ) + except anthropic.RateLimitError as error: + raise ModelRateLimitedError(str(error)) from error + except (anthropic.APIConnectionError, anthropic.APIStatusError) as error: + raise ModelUnavailableError(str(error)) from error + return "".join(block.text for block in response.content if block.type == "text") +``` + +- [ ] **Step 4: Run the tests** + +Run: `uv run pytest ord_schema/search/nl_test.py -k summary -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add ord_schema/search/nl.py ord_schema/search/nl_test.py +git commit -m "Summarize a result and let the model describe it" +``` + +--- + +### Task 6: The round trip + +**Files:** + +- Modify: `ord_schema/search/nl.py`, `ord_schema/search/nl_test.py` + +**Interfaces:** + +- Produces: `Answer(question, query, table, text)`; `ask(question, corpus, *, client=None, model=DEFAULT_MODEL, timeout_seconds=60.0) -> Answer`. + +- [ ] **Step 1: Write the failing test** + +```python +def test_ask_returns_the_query_it_ran(corpus): + # The caller needs the query to show what was actually asked, and to offer a rerun. + client = _StubClient({"where": _WHERE}, _text("Two reactions, both with pyridine.")) + result = nl.ask("solvent reactions", corpus, client=client) + assert result.query.where.op == "exists" + assert result.table.num_rows == result.table.num_rows + assert "pyridine" in result.text +``` + +Extend `_StubClient` so a canned entry that is a string becomes a text response: + +```python +def _text(value: str): + return value + + +# in _StubClient.create, before building the tool_use block: + if isinstance(payload, str): + block = types.SimpleNamespace(type="text", text=payload) + return types.SimpleNamespace(content=[block], usage=usage, stop_reason="end_turn") +``` + +- [ ] **Step 2: Run it** + +Run: `uv run pytest ord_schema/search/nl_test.py -k ask -v` +Expected: FAIL — `ask` does not exist. + +- [ ] **Step 3: Implement it** + +```python +@dataclasses.dataclass(frozen=True) +class Answer: + """What a question produced, including the query it became. + + Attributes: + question: The question as asked. + query: The query that ran, for display and for running again. + table: What the search returned. + text: A sentence or two describing the result. + """ + + question: str + query: query.Query + table: pa.Table + text: str + + +def ask( + question: str, + corpus: execute.Corpus, + *, + client: anthropic.Anthropic | None = None, + model: str = DEFAULT_MODEL, + timeout_seconds: float = 60.0, +) -> Answer: + """Answers a question against a corpus. + + Args: + question: The question, in English. + corpus: The corpus to search. + client: Anthropic client; one is built from the environment if omitted. + model: Which model translates and describes. + timeout_seconds: Passed to the search, so a slow query fails rather than hangs. + + Returns: + The query, the reactions, and a description of them. + + Raises: + MalformedQueryError: If translation does not produce a query that compiles. + ModelRateLimitedError: If the model is rate limited. + ModelUnavailableError: If the model cannot be reached. + """ + client = client if client is not None else get_client() + translated = translate(question, client=client, model=model) + table = corpus.search(translated, timeout_seconds=timeout_seconds) + return Answer( + question=question, + query=translated, + table=table, + text=answer(question, table, client=client, model=model), + ) +``` + +- [ ] **Step 4: Run the whole file** + +Run: `uv run pytest ord_schema/search/nl_test.py -v` +Expected: PASS, and no test makes a network call. + +- [ ] **Step 5: Document it** + +Add to `ord_schema/search/README.md`, under `## Usage`: + +````markdown +### Ask in English + +```python +from ord_schema.search import execute, nl + +corpus = execute.Corpus("projections/*/*.parquet", "structures/*/*.parquet") +answer = nl.ask("which reactions use pyridine as a solvent?", corpus) +print(answer.query, answer.table.num_rows, answer.text) +``` + +Needs the `nl` extra (`pip install "ord-schema[nl]"`) and `ANTHROPIC_API_KEY`. The model +cannot be constrained to emit a valid query — the grammar is recursive and both +constrained-decoding paths refuse it — so a failure is handed back once with the +compiler's own error, and a second failure raises `MalformedQueryError`. +```` + +- [ ] **Step 6: Commit** + +```bash +git add ord_schema/search/nl.py ord_schema/search/nl_test.py ord_schema/search/README.md +git commit -m "Answer a question end to end" +``` + +--- + +### Task 7: The eval harness + +**Files:** + +- Create: `ord_schema/search/nl_eval.py`, `ord_schema/search/nl_cases.yaml`, `ord_schema/search/nl_eval_test.py` + +**Interfaces:** + +- Consumes: `translate`, `execute.Corpus`. +- Produces: `EvalCase`, `load_cases(path) -> list[EvalCase]`, `run_case(case, corpus, *, client, model, repair) -> CaseResult`, `report(results) -> str`. + +- [ ] **Step 1: Write the failing test** + +```python +def test_a_case_that_returns_a_forbidden_reaction_fails(corpus): + # Scoring on reactions rather than on query shape is what lets the harness say a + # translation is wrong rather than merely differently spelled. + case = nl_eval.EvalCase( + question="solvent reactions", + must_return=["ord-aa000000000000000000000000000000"], + must_not_return=["ord-bb000000000000000000000000000000"], + ) + result = nl_eval.score(case, returned=["ord-bb000000000000000000000000000000"]) + assert not result.passed + assert "must_not_return" in result.detail +``` + +- [ ] **Step 2: Run it** + +Run: `uv run pytest ord_schema/search/nl_eval_test.py -v` +Expected: FAIL — module does not exist. + +- [ ] **Step 3: Write the harness** + +```python +class EvalCase(BaseModel): + """One question and what any correct answer to it must satisfy. + + Attributes: + question: The question, in English. + must_return: Reaction IDs any correct query returns. + must_not_return: Reaction IDs a near-miss would wrongly include. + compiles: Whether the question should translate at all; False marks a question + the grammar cannot express, which the layer must refuse rather than fudge. + """ + + question: str + must_return: list[str] = Field(default_factory=list) + must_not_return: list[str] = Field(default_factory=list) + compiles: bool = True + + +@dataclasses.dataclass(frozen=True) +class CaseResult: + """How one case came out.""" + + case: EvalCase + passed: bool + detail: str + + +def score(case: EvalCase, returned: Sequence[str]) -> CaseResult: + """Returns whether the reactions a query returned satisfy the case.""" + found = set(returned) + missing = [value for value in case.must_return if value not in found] + forbidden = [value for value in case.must_not_return if value in found] + if missing: + return CaseResult(case, False, f"must_return absent: {missing}") + if forbidden: + return CaseResult(case, False, f"must_not_return present: {forbidden}") + return CaseResult(case, True, f"{len(found)} reactions") + + +def load_cases(path: str) -> list[EvalCase]: + """Returns the cases a YAML file holds.""" + with pathlib.Path(path).open(encoding="utf-8") as handle: + return [EvalCase.model_validate(entry) for entry in yaml.safe_load(handle)] +``` + +- [ ] **Step 4: Add the runner and the report** + +```python +def run_case( + case: EvalCase, + corpus: execute.Corpus, + *, + client: anthropic.Anthropic, + model: str, + repair: bool, +) -> CaseResult: + """Translates and runs one case, returning how it scored.""" + try: + translated = nl.translate(case.question, client=client, model=model, repair=repair) + except nl.MalformedQueryError as error: + passed = not case.compiles + return CaseResult(case, passed, f"did not compile: {error}") + if not case.compiles: + return CaseResult(case, False, "compiled, but the case expects it cannot") + table = corpus.search(translated) + return score(case, table.column("reaction_id").to_pylist()) + + +def report(results: Sequence[CaseResult]) -> str: + """Returns a human-readable summary, failures first.""" + passed = sum(result.passed for result in results) + lines = [f"{passed}/{len(results)} passed"] + lines += [ + f" FAIL {result.case.question}: {result.detail}" + for result in results + if not result.passed + ] + return "\n".join(lines) +``` + +- [ ] **Step 5: Write the starting cases** + +`ord_schema/search/nl_cases.yaml`, with reaction IDs filled in by running each query by hand against the local corpus first: + +```yaml +# Each case states what any correct translation must return, never a Query literal: +# several spellings are right, and pinning one would fail a better query than the one +# that was written when the case was added. +- question: reactions using pyridine as the solvent + must_return: [] + must_not_return: [] +- question: reactions run above 350 K + must_return: [] + must_not_return: [] +- question: the ten highest-yielding reactions + must_return: [] + must_not_return: [] +``` + +- [ ] **Step 6: Run the tests** + +Run: `uv run pytest ord_schema/search/nl_eval_test.py -v` +Expected: PASS. + +- [ ] **Step 7: Take the first real measurement** + +Run, with a key in the environment and the local corpus: + +```bash +uv run python -m ord_schema.search.nl_eval --model claude-haiku-4-5 --repair +uv run python -m ord_schema.search.nl_eval --model claude-opus-5 --repair +``` + +Record both in the logbook entry rather than in a commit message: the numbers in +finding 6 came from ten ad-hoc questions and want replacing with these. + +- [ ] **Step 8: Commit** + +```bash +git add ord_schema/search/nl_eval.py ord_schema/search/nl_cases.yaml ord_schema/search/nl_eval_test.py +git commit -m "Score translations on the reactions they return" +``` + +--- + +## Self-review + +**Spec coverage.** `ask`/`translate`/`answer` — Tasks 3, 5, 6. Cached prefix — Task 3, pinned by a test. Coercion — Task 3. One repair turn — Tasks 3 and 4. Summary not table — Task 5. Module layout and the `nl` extra — Task 2. Error taxonomy — Task 2. Eval harness scoring on reactions — Task 7. The reduction gap — Task 1. Out of scope in the spec and absent here: multi-turn, a second backend, constrained decoding, text-search tuning. + +**Types.** `translate` returns `query.Query` in Tasks 3, 6, and 7. `Answer.query` is that same type. `summarize` and `answer` both take `pa.Table`, which is what `Corpus.search` returns. `Reduction` is referenced by `Order.key` and `Measure.path` only. + +**Known gaps, deliberately left to the executor.** The eval cases in Task 7 ship with empty `must_return` lists, because the IDs have to come from running each query against the local corpus — filling them in is step 5's work, and inventing IDs here would be worse than leaving them empty. Task 1's step 6 needs the local projections; it is a check, not a test, and does not run in CI. diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_bisect.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_bisect.py new file mode 100644 index 0000000..308697e --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_bisect.py @@ -0,0 +1,60 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Which part of the grammar makes the compiled decoding grammar too large?""" + +import json +import pathlib +import sys + +import anthropic +from anthropic.lib._parse._transform import transform_schema +from pydantic import BaseModel + +sys.path.insert(0, str(pathlib.Path(__file__).parent)) +from stratify import stratify # noqa: E402 + +from ord_schema.search import query # noqa: E402 + + +class Tiny(BaseModel): + name: str + count: int + + +def sized(schema: dict) -> str: + return f"{len(json.dumps(schema)) // 4:,} ~tokens" + + +client = anthropic.Anthropic() +cases = [ + ("a tiny two-field object", transform_schema(Tiny.model_json_schema())), + ("one comparison predicate", transform_schema(query.Comparison.model_json_schema())), + ("one quantifier", transform_schema(query.Quantifier.model_json_schema())), + ("Query, stratified depth 0", transform_schema(stratify(query.Query.model_json_schema(), 0))), + ("Query, stratified depth 1", transform_schema(stratify(query.Query.model_json_schema(), 1))), +] +for label, schema in cases: + try: + client.messages.create( + model="claude-haiku-4-5", + max_tokens=512, + messages=[{"role": "user", "content": "reactions above 350 K"}], + output_config={"format": {"type": "json_schema", "schema": schema}}, + ) + except anthropic.APIStatusError as error: + message = json.loads(error.response.text)["error"]["message"] + print(f" {label:32} {sized(schema):>16} REFUSED: {message[:80]}") + else: + print(f" {label:32} {sized(schema):>16} accepted") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_flat_union.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_flat_union.py new file mode 100644 index 0000000..9171b67 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_flat_union.py @@ -0,0 +1,113 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Can a flattened predicate -- one object per level, path as an enum -- fit the budget? + +The union of eight predicate variants is what multiplies the state machine, so this +collapses them into one object whose `op` says which shape it is. Paths become an enum, +which is the only way the decoder can stop a model inventing `identifiers[*]`. +""" + +import json +import sys + +import anthropic +import pyarrow as pa + +from ord_schema.artifacts import projection + +client = anthropic.Anthropic() + + +def leaf_paths(schema: pa.Schema) -> list[str]: + """Returns every scalar column path in the projection, dotted.""" + found: list[str] = [] + + def walk(field: pa.Field, prefix: str) -> None: + path = f"{prefix}.{field.name}" if prefix else field.name + dtype = field.type + while pa.types.is_list(dtype) or pa.types.is_map(dtype): + dtype = dtype.item_type if pa.types.is_map(dtype) else dtype.value_type + if pa.types.is_struct(dtype): + for child in dtype: + walk(child, path) + else: + found.append(path) + + for field in schema: + walk(field, "") + return found + + +PATHS = leaf_paths(projection.SCHEMA) +print(f"projection leaves: {len(PATHS)}") + + +def predicate(level: int, paths: list[str]) -> dict: + """Returns a flat predicate object whose clauses are one level shallower.""" + node: dict = { + "type": "object", + "additionalProperties": False, + "properties": { + "op": { + "type": "string", + "enum": [ + "and", "or", "not", "exists", "forall", + "eq", "ne", "lt", "le", "gt", "ge", + "contains", "starts_with", "ends_with", + "is_null", "not_null", "substructure", "similarity", + ], + }, + "path": {"anyOf": [{"type": "string", "enum": paths}, {"type": "null"}]}, + "literal": {"anyOf": [{"type": "string"}, {"type": "number"}, + {"type": "boolean"}, {"type": "null"}]}, + "compound": {"anyOf": [{"type": "string"}, {"type": "null"}]}, + "smarts": {"anyOf": [{"type": "string"}, {"type": "null"}]}, + }, + "required": ["op", "path", "literal", "compound", "smarts"], + } + if level > 0: + node["properties"]["clauses"] = { + "anyOf": [{"type": "array", "items": predicate(level - 1, paths)}, + {"type": "null"}] + } + node["required"].append("clauses") + return node + + +for count in (0, 40, 120, len(PATHS)): + paths = PATHS[:count] if count else [""] + for depth in (2, 3, 4): + schema = { + "type": "object", + "additionalProperties": False, + "properties": {"where": predicate(depth, paths)}, + "required": ["where"], + } + if not count: # free-form path, for comparison with the enum forms + schema = json.loads(json.dumps(schema).replace( + '{"type": "string", "enum": [""]}', '{"type": "string"}')) + size = f"{len(json.dumps(schema)) // 4:,} ~tok" + label = f"paths={count or 'free':>4} depth={depth}" + try: + client.messages.create( + model="claude-haiku-4-5", max_tokens=256, + messages=[{"role": "user", "content": "reactions above 350 K"}], + output_config={"format": {"type": "json_schema", "schema": schema}}, + ) + except anthropic.APIStatusError as error: + message = json.loads(error.response.text)["error"]["message"] + print(f" {label} {size:>12} REFUSED: {message[:52]}") + else: + print(f" {label} {size:>12} ACCEPTED") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion.py new file mode 100644 index 0000000..feebe96 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion.py @@ -0,0 +1,103 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""What does the SDK send for a recursive output_format, and does it object first? + +Captures the request body with a mock transport rather than a live call, so the +client-side half of the question is settled without a key: whether the SDK accepts a +self-referencing pydantic model at all, and what JSON Schema it puts on the wire. +""" + +import json + +import anthropic +import httpx + +from ord_schema.search import query + +CANNED = { + "id": "msg_probe", + "type": "message", + "role": "assistant", + "model": "claude-opus-5", + "content": [{"type": "text", "text": "{}"}], + "stop_reason": "end_turn", + "usage": {"input_tokens": 1, "output_tokens": 1}, +} +sent = {} + + +def handler(request: httpx.Request) -> httpx.Response: + sent["body"] = json.loads(request.content) + return httpx.Response(200, json=CANNED) + + +client = anthropic.Anthropic( + api_key="probe-not-a-real-key", + http_client=httpx.Client(transport=httpx.MockTransport(handler)), +) + +print("anthropic SDK:", anthropic.__version__) +for label, call in ( + ( + "messages.parse(output_format=Query)", + lambda: client.messages.parse( + model="claude-opus-5", + max_tokens=1024, + messages=[{"role": "user", "content": "pyridine as solvent"}], + output_format=query.Query, + ), + ), + ( + "messages.create(tools=[strict build_query])", + lambda: client.messages.create( + model="claude-opus-5", + max_tokens=1024, + messages=[{"role": "user", "content": "pyridine as solvent"}], + tools=[ + { + "name": "build_query", + "description": "Build a search query.", + "strict": True, + "input_schema": query.Query.model_json_schema(), + } + ], + tool_choice={"type": "tool", "name": "build_query"}, + ), + ), +): + sent.clear() + print(f"\n=== {label}") + try: + call() + except Exception as error: # noqa: BLE001 -- the probe reports whatever it hits. + print(f" raised {type(error).__name__}: {str(error)[:200]}") + body = sent.get("body") + if body is None: + print(" nothing reached the wire") + continue + schema = None + if "output_config" in body: + schema = body["output_config"].get("format", {}).get("schema") + print(" output_config.format keys:", sorted(body["output_config"]["format"])) + elif "tools" in body: + schema = body["tools"][0].get("input_schema") + print(" tool keys:", sorted(body["tools"][0])) + if schema is None: + print(" no schema on the wire") + continue + text = json.dumps(schema) + print(f" schema on the wire: {len(text)} chars, $defs={len(schema.get('$defs', {}))}") + print(f" contains $ref: {'$ref' in text}, refs={text.count('#/$defs/')}") + print(f" additionalProperties present: {text.count('additionalProperties')}") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion_live.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion_live.py new file mode 100644 index 0000000..419290c --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion_live.py @@ -0,0 +1,57 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Does the API accept a recursive json_schema output format? Needs a real key. + + ANTHROPIC_API_KEY=sk-... uv run --with anthropic python probe_recursion_live.py + +Prints the validated Query the model produced, or the error the server returned. +""" + +import json + +import anthropic + +from ord_schema.search import query, schema + +SYSTEM = ( + "You translate chemistry questions into ORD search queries. The corpus schema, " + "as an indented type tree in DuckDB's types:\n\n" + schema.describe() +) + +client = anthropic.Anthropic() +try: + response = client.messages.parse( + model="claude-opus-5", + max_tokens=2048, + system=[{"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}}], + messages=[ + { + "role": "user", + "content": "reactions using pyridine as the solvent with a yield above 50%", + } + ], + output_format=query.Query, + ) +except anthropic.APIStatusError as error: + print(f"server refused it: {error.status_code} {str(error)[:400]}") +else: + print("accepted. parsed_output:") + print(json.dumps(response.parsed_output.model_dump(exclude_none=True), indent=2)) + usage = response.usage + print( + f"\ninput={usage.input_tokens} " + f"cache_write={usage.cache_creation_input_tokens} " + f"cache_read={usage.cache_read_input_tokens} output={usage.output_tokens}" + ) diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_repair.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_repair.py new file mode 100644 index 0000000..2220dd4 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_repair.py @@ -0,0 +1,130 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Does one repair turn carry a cheap model to the accuracy of an expensive one? + +The compiler's errors name the offending path and suggest a real one, so a failed +translation is handed straight back. Scores first-try and after-repair separately. +""" + +import json +import sys + +import anthropic + +from ord_schema.search import query, schema + +QUESTIONS = [ + "reactions using pyridine as the solvent", + "reactions using pyridine as the solvent with a yield above 50%", + "reactions run above 350 K", + "reactions where every input component is a liquid", + "average yield by reaction temperature, for reactions with a boronic acid", + "reactions that make aspirin", + "the ten highest-yielding Suzuki couplings", + "reactions with no solvent at all", + "how many reactions use palladium catalysts", + "reactions stirred for more than an hour at above 100 C", +] +SYSTEM = [ + { + "type": "text", + "text": ( + "You translate chemistry questions into ORD search queries by calling " + "build_query. The corpus schema, as an indented type tree in DuckDB's " + "types:\n\n" + schema.describe() + ), + "cache_control": {"type": "ephemeral"}, + } +] +TOOL = { + "name": "build_query", + "description": "Build an ORD search query from the user's question.", + "input_schema": query.Query.model_json_schema(), +} +MODEL = sys.argv[1] if len(sys.argv) > 1 else "claude-haiku-4-5" + + +def coerce(value): + """Returns the input with any JSON-encoded string values parsed back to objects.""" + if isinstance(value, str): + try: + return coerce(json.loads(value)) + except (json.JSONDecodeError, TypeError): + return value + if isinstance(value, dict): + return {key: coerce(item) for key, item in value.items()} + if isinstance(value, list): + return [coerce(item) for item in value] + return value + + +def check(raw) -> str | None: + """Returns the error a translation fails with, or None if it compiles.""" + try: + query.compile_query(query.Query.model_validate(coerce(raw))) + except Exception as error: # noqa: BLE001 -- whatever it fails with is the message. + return str(error) + return None + + +client = anthropic.Anthropic() +first_try = repaired = 0 +cost = {"cache_read": 0, "input": 0, "output": 0} +for question in QUESTIONS: + messages: list = [{"role": "user", "content": question}] + response = client.messages.create( + model=MODEL, max_tokens=2048, system=SYSTEM, messages=messages, + tools=[TOOL], tool_choice={"type": "tool", "name": "build_query"}, + ) + cost["cache_read"] += response.usage.cache_read_input_tokens or 0 + cost["input"] += response.usage.input_tokens + cost["output"] += response.usage.output_tokens + block = next(b for b in response.content if b.type == "tool_use") + error = check(block.input) + if error is None: + first_try += 1 + repaired += 1 + print(f" {question[:48]:50} ok") + continue + messages += [ + {"role": "assistant", "content": response.content}, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": block.id, + "is_error": True, + "content": f"That query was rejected: {error}. Call build_query again with it fixed.", + } + ], + }, + ] + retry = client.messages.create( + model=MODEL, max_tokens=2048, system=SYSTEM, messages=messages, + tools=[TOOL], tool_choice={"type": "tool", "name": "build_query"}, + ) + cost["cache_read"] += retry.usage.cache_read_input_tokens or 0 + cost["input"] += retry.usage.input_tokens + cost["output"] += retry.usage.output_tokens + retry_block = next(b for b in retry.content if b.type == "tool_use") + second = check(retry_block.input) + if second is None: + repaired += 1 + print(f" {question[:48]:50} repaired ({error.split(';')[0][:44]})") + else: + print(f" {question[:48]:50} STILL FAILS: {second[:60]}") +print(f"\n{MODEL}: first try {first_try}/{len(QUESTIONS)}, after repair {repaired}/{len(QUESTIONS)}") +print(f"tokens: {cost}") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_required.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_required.py new file mode 100644 index 0000000..b580fb8 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_required.py @@ -0,0 +1,46 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Does requiring every property bring the grammar under the compiler's budget?""" + +import copy +import json +import pathlib +import sys + +import anthropic + +sys.path.insert(0, str(pathlib.Path(__file__).parent)) +from require_all import require_all # noqa: E402 +from stratify import stratify # noqa: E402 + +from ord_schema.search import query # noqa: E402 + +client = anthropic.Anthropic() +base = query.Query.model_json_schema() +for depth in range(0, 6): + schema = require_all(stratify(copy.deepcopy(base), depth), strip_descriptions=True) + size = f"{len(json.dumps(schema)) // 4:,} ~tokens" + try: + client.messages.create( + model="claude-haiku-4-5", + max_tokens=512, + messages=[{"role": "user", "content": "reactions above 350 K"}], + output_config={"format": {"type": "json_schema", "schema": schema}}, + ) + except anthropic.APIStatusError as error: + message = json.loads(error.response.text)["error"]["message"] + print(f" depth {depth} {size:>14} REFUSED: {message[:78]}") + else: + print(f" depth {depth} {size:>14} ACCEPTED") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_shapes.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_shapes.py new file mode 100644 index 0000000..387845f --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_shapes.py @@ -0,0 +1,110 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""How often does a forced tool call over the recursive grammar parse as written? + +Runs a handful of questions through two models and scores three ways: as returned, +after coercing JSON-encoded strings back to objects, and whether the compiler accepts +the result. The gap between the first two is what a strict schema would have bought. +""" + +import json + +import anthropic + +from ord_schema.search import query, schema + +QUESTIONS = [ + "reactions using pyridine as the solvent", + "reactions using pyridine as the solvent with a yield above 50%", + "reactions run above 350 K", + "reactions where every input component is a liquid", + "average yield by reaction temperature, for reactions with a boronic acid", +] +SYSTEM = [ + { + "type": "text", + "text": ( + "You translate chemistry questions into ORD search queries. Emit the query " + "by calling build_query. The corpus schema, as an indented type tree in " + "DuckDB's types:\n\n" + schema.describe() + ), + "cache_control": {"type": "ephemeral"}, + } +] +TOOL = { + "name": "build_query", + "description": "Build an ORD search query from the user's question.", + "input_schema": query.Query.model_json_schema(), +} + + +def coerce(value): + """Returns the input with any JSON-encoded string values parsed back to objects.""" + if isinstance(value, str): + try: + return coerce(json.loads(value)) + except (json.JSONDecodeError, TypeError): + return value + if isinstance(value, dict): + return {key: coerce(item) for key, item in value.items()} + if isinstance(value, list): + return [coerce(item) for item in value] + return value + + +client = anthropic.Anthropic() +for model in ("claude-opus-5", "claude-haiku-4-5"): + print(f"\n=== {model}") + tallies = {"as written": 0, "coerced": 0, "compiles": 0} + cost = {"cache_write": 0, "cache_read": 0, "input": 0, "output": 0} + for question in QUESTIONS: + response = client.messages.create( + model=model, + max_tokens=2048, + system=SYSTEM, + messages=[{"role": "user", "content": question}], + tools=[TOOL], + tool_choice={"type": "tool", "name": "build_query"}, + ) + usage = response.usage + cost["cache_write"] += usage.cache_creation_input_tokens or 0 + cost["cache_read"] += usage.cache_read_input_tokens or 0 + cost["input"] += usage.input_tokens + cost["output"] += usage.output_tokens + raw = next(b for b in response.content if b.type == "tool_use").input + note = [] + try: + query.Query.model_validate(raw) + except Exception: # noqa: BLE001 -- scoring, not handling. + note.append("as-written FAILED") + else: + tallies["as written"] += 1 + note.append("as-written ok") + try: + parsed = query.Query.model_validate(coerce(raw)) + except Exception as error: # noqa: BLE001 + note.append(f"coerced FAILED: {str(error)[:60]}") + else: + tallies["coerced"] += 1 + try: + query.compile_query(parsed) + except Exception as error: # noqa: BLE001 + note.append(f"compile FAILED: {str(error)[:80]}") + else: + tallies["compiles"] += 1 + note.append("compiles") + print(f" {question[:52]:54} {' | '.join(note)}") + print(f" tallies (of {len(QUESTIONS)}): {tallies}") + print(f" tokens: {cost}") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_stratified.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_stratified.py new file mode 100644 index 0000000..1617419 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_stratified.py @@ -0,0 +1,93 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Does a stratified grammar unlock structured outputs, and can a cheap model hit it? + +Same five questions as the tool-call probe, but the model is handed an acyclic schema +and constrained by output_config.format rather than asked to call a tool. +""" + +import json +import pathlib +import sys + +import anthropic +from anthropic.lib._parse._transform import transform_schema + +sys.path.insert(0, str(pathlib.Path(__file__).parent)) +from stratify import stratify # noqa: E402 + +from ord_schema.search import query, schema # noqa: E402 + +QUESTIONS = [ + "reactions using pyridine as the solvent", + "reactions using pyridine as the solvent with a yield above 50%", + "reactions run above 350 K", + "reactions where every input component is a liquid", + "average yield by reaction temperature, for reactions with a boronic acid", +] +SYSTEM = [ + { + "type": "text", + "text": ( + "You translate chemistry questions into ORD search queries. The corpus " + "schema, as an indented type tree in DuckDB's types:\n\n" + schema.describe() + ), + "cache_control": {"type": "ephemeral"}, + } +] +DEPTH = int(sys.argv[1]) if len(sys.argv) > 1 else 4 +MODEL = sys.argv[2] if len(sys.argv) > 2 else "claude-haiku-4-5" +STRATIFIED = transform_schema(stratify(query.Query.model_json_schema(), DEPTH)) + +client = anthropic.Anthropic() +print(f"=== {MODEL}, stratified depth {DEPTH} " + f"({len(json.dumps(STRATIFIED))//4:,} ~tokens)") +tallies = {"parses": 0, "compiles": 0} +cost = {"cache_write": 0, "cache_read": 0, "input": 0, "output": 0} +for question in QUESTIONS: + try: + response = client.messages.create( + model=MODEL, + max_tokens=2048, + system=SYSTEM, + messages=[{"role": "user", "content": question}], + output_config={"format": {"type": "json_schema", "schema": STRATIFIED}}, + ) + except anthropic.APIStatusError as error: + print(f" refused: {error.status_code} {str(error)[:300]}") + break + usage = response.usage + cost["cache_write"] += usage.cache_creation_input_tokens or 0 + cost["cache_read"] += usage.cache_read_input_tokens or 0 + cost["input"] += usage.input_tokens + cost["output"] += usage.output_tokens + text = next(b.text for b in response.content if b.type == "text") + note = [] + try: + parsed = query.Query.model_validate(json.loads(text)) + except Exception as error: # noqa: BLE001 -- scoring, not handling. + note.append(f"parse FAILED: {str(error)[:90]}") + else: + tallies["parses"] += 1 + try: + query.compile_query(parsed) + except Exception as error: # noqa: BLE001 + note.append(f"compile FAILED: {str(error)[:90]}") + else: + tallies["compiles"] += 1 + note.append("compiles") + print(f" {question[:52]:54} {' | '.join(note)}") +print(f" tallies (of {len(QUESTIONS)}): {tallies}") +print(f" tokens: {cost}") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_strict_tool.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_strict_tool.py new file mode 100644 index 0000000..eb52917 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_strict_tool.py @@ -0,0 +1,52 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Does the strict-tool path have a bigger grammar budget than output_config.format?""" + +import json +import pathlib +import sys + +import anthropic +from anthropic.lib._parse._transform import transform_schema + +sys.path.insert(0, str(pathlib.Path(__file__).parent)) +from stratify import stratify # noqa: E402 + +from ord_schema.search import query # noqa: E402 + +client = anthropic.Anthropic() +base = query.Query.model_json_schema() +for depth in (0, 1, 2, 4): + schema = transform_schema(stratify(base, depth)) + tool = { + "name": "build_query", + "description": "Build an ORD search query.", + "strict": True, + "input_schema": schema, + } + size = f"{len(json.dumps(schema)) // 4:,} ~tokens" + try: + client.messages.create( + model="claude-haiku-4-5", + max_tokens=512, + messages=[{"role": "user", "content": "reactions above 350 K"}], + tools=[tool], + tool_choice={"type": "tool", "name": "build_query"}, + ) + except anthropic.APIStatusError as error: + message = json.loads(error.response.text)["error"]["message"] + print(f" strict tool, depth {depth} {size:>16} REFUSED: {message[:70]}") + else: + print(f" strict tool, depth {depth} {size:>16} accepted") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_tool_recursion.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_tool_recursion.py new file mode 100644 index 0000000..e39180b --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_tool_recursion.py @@ -0,0 +1,80 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Do tool input schemas accept the recursion that output_config.format refuses? + +Three shapes, cheapest question that still needs a nested predicate: a forced tool call +over the grammar as pydantic writes it, the same with strict on and the SDK's own +closing transform applied, and -- for reference -- what the model actually produces. +""" + +import json + +import anthropic +from anthropic.lib._parse._transform import transform_schema + +from ord_schema.search import query, schema + +QUESTION = "reactions using pyridine as the solvent with a yield above 50%" +SYSTEM = [ + { + "type": "text", + "text": ( + "You translate chemistry questions into ORD search queries. The corpus " + "schema, as an indented type tree in DuckDB's types:\n\n" + schema.describe() + ), + "cache_control": {"type": "ephemeral"}, + } +] +RAW = query.Query.model_json_schema() + +client = anthropic.Anthropic() +for label, input_schema, strict in ( + ("tool, as pydantic writes it", RAW, False), + ("tool, strict + closed", transform_schema(json.loads(json.dumps(RAW))), True), +): + tool: dict = { + "name": "build_query", + "description": "Build an ORD search query from the user's question.", + "input_schema": input_schema, + } + if strict: + tool["strict"] = True + print(f"\n=== {label}") + try: + response = client.messages.create( + model="claude-opus-5", + max_tokens=2048, + system=SYSTEM, + messages=[{"role": "user", "content": QUESTION}], + tools=[tool], + tool_choice={"type": "tool", "name": "build_query"}, + ) + except anthropic.APIStatusError as error: + print(f" refused: {error.status_code} {str(error)[:220]}") + continue + block = next(b for b in response.content if b.type == "tool_use") + print(" accepted. tool_use.input:") + print(json.dumps(block.input, indent=2)[:900]) + try: + parsed = query.Query.model_validate(block.input) + except Exception as error: # noqa: BLE001 -- the probe reports whatever it hits. + print(f" pydantic REJECTED it: {type(error).__name__}: {str(error)[:200]}") + else: + print(f" pydantic accepted it: {parsed.where.op}") + usage = response.usage + print( + f" input={usage.input_tokens} cache_write={usage.cache_creation_input_tokens} " + f"cache_read={usage.cache_read_input_tokens} output={usage.output_tokens}" + ) diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/require_all.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/require_all.py new file mode 100644 index 0000000..83c9b6c --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/require_all.py @@ -0,0 +1,105 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Makes every property required, expressing optionality as an explicit null instead. + +Constrained decoding compiles a schema into a state machine, and an optional property +doubles it: the machine has to accept the object with and without that key. Requiring +every key and letting the value be null costs one token per absent field and leaves the +machine linear in the number of properties. +""" + +import copy + +# Keywords a decoder rejects or ignores. Dropping them costs nothing: the response is +# validated by the pydantic models afterwards, which enforce all of them anyway. +_UNSUPPORTED = frozenset( + { + "minimum", + "maximum", + "exclusiveMinimum", + "exclusiveMaximum", + "multipleOf", + "minLength", + "maxLength", + "pattern", + "maxItems", + "uniqueItems", + "default", + } +) + + +def require_all(node, strip_descriptions: bool = False): + """Returns node with every property required and nullable where it was optional.""" + if isinstance(node, list): + return [require_all(item, strip_descriptions) for item in node] + if not isinstance(node, dict): + return node + out = {} + for key, value in node.items(): + if strip_descriptions and key in ("description", "title", "examples"): + continue + # A decoder takes anyOf but not oneOf, and the discriminator that rides with it + # is pydantic's business rather than the model's: validation still happens here. + if key == "discriminator" or key in _UNSUPPORTED: + continue + out["anyOf" if key == "oneOf" else key] = require_all(value, strip_descriptions) + properties = out.get("properties") + if isinstance(properties, dict) and properties: + required = set(out.get("required", [])) + for name, subschema in properties.items(): + if name in required: + continue + branches = subschema.get("anyOf") if isinstance(subschema, dict) else None + if branches and any(b.get("type") == "null" for b in branches): + continue + properties[name] = {"anyOf": [subschema, {"type": "null"}]} + out["required"] = sorted(properties) + out["additionalProperties"] = False + return out + + +def count_optional(node, seen=None) -> int: + """Returns how many properties a schema leaves optional.""" + total = 0 + if isinstance(node, dict): + properties = node.get("properties") + if isinstance(properties, dict): + total += len(set(properties) - set(node.get("required", []))) + for value in node.values(): + total += count_optional(value) + elif isinstance(node, list): + for item in node: + total += count_optional(item) + return total + + +if __name__ == "__main__": + import json + import sys + + sys.path.insert(0, str(pathlib.Path(__file__).parent)) + from stratify import stratify + + from ord_schema.search import query + + base = query.Query.model_json_schema() + print(f"{'depth':>6} {'optional before':>16} {'after':>6} {'~tokens':>9}") + for depth in range(0, 5): + built = stratify(copy.deepcopy(base), depth) + before = count_optional(built) + required = require_all(built, strip_descriptions=True) + text = json.dumps(required) + print(f"{depth:>6} {before:>16} {count_optional(required):>6} {len(text)//4:>9,}") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/stratify.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/stratify.py new file mode 100644 index 0000000..2ff2ce5 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/stratify.py @@ -0,0 +1,109 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Turns the recursive grammar into an acyclic one by stratifying it into levels. + +The API refuses a schema whose definitions reference each other in a cycle, which is +what a predicate tree is. Stratifying keeps every definition but numbers it: a level-k +predicate's clauses are level-(k-1) predicates, and level 0 holds only the leaves. The +refs stay refs -- nothing is inlined, so size grows by a level rather than by a power -- +and the result validates into the same recursive pydantic models. +""" + +import copy +import json + + +def _referenced(node, found): + """Collects the definition names ``node`` references.""" + if isinstance(node, dict): + ref = node.get("$ref") + if ref: + found.add(ref.split("/")[-1]) + for value in node.values(): + _referenced(value, found) + elif isinstance(node, list): + for item in node: + _referenced(item, found) + return found + + +def _retarget(node, level, levelled): + """Returns node with refs pointing at the next level down.""" + if isinstance(node, dict): + ref = node.get("$ref") + if ref: + name = ref.split("/")[-1] + if name not in levelled: + return {"$ref": f"#/$defs/{name}"} + if level <= 0: + return None + return {"$ref": f"#/$defs/{name}_L{level - 1}"} + out = {} + for key, value in node.items(): + result = _retarget(value, level, levelled) + if result is None: + if key in ("anyOf", "items", "properties"): + return None + continue + if isinstance(result, list) and not result: + return None + out[key] = result + if "anyOf" in out and not out["anyOf"]: + return None + return out + if isinstance(node, list): + kept = [_retarget(item, level, levelled) for item in node] + return [item for item in kept if item is not None] + return node + + +def stratify(schema: dict, depth: int) -> dict: + """Returns an acyclic schema whose predicates nest at most ``depth`` levels. + + Args: + schema: A JSON Schema with a recursive ``$defs`` section. + depth: How many levels of nesting to allow; level 0 holds only leaves. + + Returns: + A schema with no reference cycle, its definitions suffixed by level. + """ + schema = copy.deepcopy(schema) + defs = schema.pop("$defs", {}) + # A definition that reaches another definition is one a level has to renumber; + # leaves reference nothing, so one copy of each serves every level. + levelled = {name for name, node in defs.items() if _referenced(node, set())} + out: dict = {} + for name, node in defs.items(): + if name not in levelled: + out[name] = node + continue + for level in range(depth + 1): + built = _retarget(copy.deepcopy(node), level, levelled) + if built is not None: + out[f"{name}_L{level}"] = built + root = _retarget(schema, depth, levelled) + root["$defs"] = out + return root + + +if __name__ == "__main__": + from ord_schema.search import query + + base = query.Query.model_json_schema() + print(f"{'depth':>6} {'chars':>9} {'~tokens':>8} {'defs':>6}") + for depth in range(1, 7): + built = stratify(base, depth) + text = json.dumps(built) + print(f"{depth:>6} {len(text):>9,} {len(text)//4:>8,} {len(built['$defs']):>6}")