From 5198a6beefccc5719b04165d06b2174e30c3abf1 Mon Sep 17 00:00:00 2001 From: Steven Kearnes Date: Mon, 17 Aug 2026 21:40:59 -0400 Subject: [PATCH 1/3] Record what constrains a natural-language layer, and the design it forces Constrained decoding cannot carry the search grammar: circular refs are refused, the compiled grammar has a size budget, oneOf is unsupported, and union-typed parameters are capped. Stratifying into acyclic levels, requiring every property, and flattening the predicate union each moved a wall, and the best that fits is a depth-2 tree -- which cannot express the correlation the pivots exist to serve. So translation is unconstrained generation checked afterwards, which the measurements say is fine: Opus 5 compiles 9/10 first try, Haiku 4.5 gets 7/10 with one repair turn at 5.6x less. The design beside the entry builds on that, and starts by closing a gap the probing exposed -- an aggregate cannot reach a value under a repeated level, so "the ten highest-yielding reactions" is unwritable and both models tried anyway. Co-Authored-By: Claude Opus 5 (1M context) --- .../README.md | 200 ++++++++++++++++++ .../assets/README.md | 49 +++++ .../assets/nl-search-design.md | 182 ++++++++++++++++ .../assets/probe_bisect.py | 60 ++++++ .../assets/probe_flat_union.py | 113 ++++++++++ .../assets/probe_recursion.py | 103 +++++++++ .../assets/probe_recursion_live.py | 57 +++++ .../assets/probe_repair.py | 130 ++++++++++++ .../assets/probe_required.py | 46 ++++ .../assets/probe_shapes.py | 110 ++++++++++ .../assets/probe_stratified.py | 93 ++++++++ .../assets/probe_strict_tool.py | 52 +++++ .../assets/probe_tool_recursion.py | 80 +++++++ .../assets/require_all.py | 105 +++++++++ .../assets/stratify.py | 109 ++++++++++ 15 files changed, 1489 insertions(+) create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/README.md create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-design.md create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_bisect.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_flat_union.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion_live.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_repair.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_required.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_shapes.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_stratified.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_strict_tool.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_tool_recursion.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/require_all.py create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/stratify.py diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md new file mode 100644 index 0000000..04d575f --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md @@ -0,0 +1,200 @@ +# What constrains a natural-language layer over the search grammar + +- **Date:** 2026-08-17 +- **Author:** Steven Kearnes +- **Status:** final (the design that follows from it is [beside this entry](assets/nl-search-design.md)) +- **Tags:** ord-schema, agents, search, natural-language, structured-outputs, anthropic, cost +- **License:** [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/) + +## Question + +`ord_schema.search` gives a model a validated `Query` grammar to write against instead of +SQL. The layer that turns a chemist's sentence into one of those queries has never +shipped: it exists on the unmerged `nl-query-backend` branch, written against a flat IR +that predates the grammar, and the `/ask` deployment it served has since been moved out +from under it. + +Rebuilding it raises one question that decides the whole design. Can the model be +*constrained* to emit a valid `Query` — structured outputs or a strict tool, where the +decoder itself refuses anything off-grammar — or does the layer have to accept whatever +comes back and check it afterwards? And with that settled: how good is a cheap model, and +what does a query cost? + +## Summary + +**Constrained decoding cannot carry this grammar.** Four separate walls, each hit +directly against the API rather than inferred: circular references are rejected outright, +the compiled grammar has a size budget, `oneOf` is unsupported, and there is a cap on +union-typed parameters. Stratifying the grammar into acyclic levels, requiring every +property, stripping descriptions and numeric constraints, and flattening the predicate +union each moved a wall — and the best that fits is a **depth-2** predicate tree, which +cannot express the nested correlation the pivots exist to serve. + +So translation is unconstrained generation, validated afterwards. That is fine: over ten +questions, **Opus 5 compiled 9/10 first try**, and **Haiku 4.5 reached 7/10 with one +repair turn** at **5.6× lower cost**. Both models get the shape wrong in the same +harmless way — they return the predicate tree as a JSON *string* — which a coercing parse +fixes deterministically. + +The probing also turned up a **gap in the grammar itself**: an aggregate or ordering key +cannot reach a value under a repeated level, so *"the ten highest-yielding reactions"* is +unwritable. Both models tried it, and repair does not help, because nothing they could +have written would have worked. + +## Method + +Every claim here is a live call against the Messages API, using the key the interface +deployment already holds (`ord-interface-anthropic-api-key`). The probes are in +[`assets/`](assets/README.md); the schema transforms they use — stratification and the +all-required rewrite — are the interesting half and are worth reading before repeating +this work. + +Ten questions were used throughout, from single-predicate (`reactions run above 350 K`) +to correlated (`pyridine as the solvent with a yield above 50%`) to aggregate (`the ten +highest-yielding Suzuki couplings`). A translation is scored on whether it validates +against the pydantic models and then compiles, which is a stricter bar than "looks right" +and a weaker one than "returns the right reactions" — result-level scoring wants an eval +set, which the design proposes and this entry does not have. + +## Findings + +### 1. The recursive grammar is refused by every constrained path + +`Query`'s predicate tree is recursive: `and`/`or` hold clauses, `not` holds a clause, and +a quantifier's body may hold another quantifier. Pydantic renders that as 12 definitions +with 134 internal references, 2,431 tokens. + +| path | result | +| --- | --- | +| `output_config.format` (structured outputs) | `Circular reference detected in schema definitions: And -> And` | +| tool with `strict: true` | same refusal | +| tool without `strict` | **accepted** | + +The SDK is not the obstacle: it sends the recursion faithfully, and its transform has +deliberate handling for `$defs` and root-level `$ref`. The refusal is server-side, and +the [structured outputs documentation](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) +confirms recursive schemas are unsupported. + +### 2. Removing the recursion does not help, because the wall behind it is smaller + +The grammar can be made acyclic without changing the IR: stratify it, so a level-*k* +predicate's clauses are level-*(k-1)* predicates and level 0 holds only leaves. Nothing is +inlined, so size grows by about **1,400 tokens per level** rather than by a power of four +— naive inlining reaches 331,086 tokens at depth 5, stratification 9,274. + +That clears the cycle and immediately hits the next wall: + +| depth | schema | `output_config.format` | strict tool | +| --- | --- | --- | --- | +| 0 | 2,425 tok | accepted | too large | +| 1 | 3,944 tok | too large | too large | +| 2 | 5,415 tok | too large | too large | +| 4 | 8,359 tok | too large | too large | + +Depth 0 is a single leaf predicate: no boolean, no quantifier. The strict-tool path is +tighter still and fits nothing. + +### 3. The budget is union-typed parameters, not tokens + +A [reported limit](https://github.com/anthropics/anthropic-sdk-python/issues/1185) says +optional properties roughly double the compiled state machine. Requiring every property +and expressing optionality as an explicit null takes the grammar from 13–17 optional +properties to zero and shrinks it by a third — and then two more walls appear in turn: +`oneOf` is unsupported (pydantic emits it for the discriminated union; rewriting to +`anyOf` is safe, since the models validate afterwards), and numeric constraints like +`exclusiveMinimum` are rejected. + +Past those, depth 0 fits at 1,000 tokens and depth 1 does not, at 1,653. The error +changes to *"too many parameters with union type"*, which names the real budget: every +nullable field is a union, and each level multiplies them. + +### 4. What does fit is a flat predicate at depth 2 — with every path enumerated + +Collapsing the eight predicate variants into one object whose `op` selects the shape, and +making `path` an enum, buys the most room. The enum turns out to be nearly free: + +| paths enumerated | depth 2 | depth 3 | depth 4 | +| --- | --- | --- | --- | +| free-form string | accepted | refused | refused | +| 40 | accepted | refused | refused | +| 120 | accepted | refused | refused | +| **all 431 scalar paths** | **accepted** (15,240 tok) | refused | refused | + +The enumeration is the 431 scalar paths `probe_flat_union.py` walks to, which skips the +key side of a map; the schema rendering counts 442 leaves and 537 lines because it carries +those too. + +This is worth remembering rather than building. Enumerating the paths makes an invented +column *structurally impossible*, and invented columns are the cheap model's main failure +mode. But depth 2 cannot nest a quantifier inside a quantifier, so correlated questions — +the ones the pivoted element index exists to answer — fall outside it. + +### 5. Unconstrained, both models write the tree as a string + +Given a non-strict forced tool call, the model reliably returns: + +```json +{"where": "{\"op\": \"and\", \"clauses\": [ ... ]}"} +``` + +The predicate tree arrives JSON-encoded inside a string, and pydantic rejects it. Opus did +this on 4 of 5 questions, Haiku on 5 of 5. Parsing string values back to objects before +validating takes both to 5/5 — this is the one thing a strict schema would have bought, +and it costs about ten lines to do without one. + +### 6. Opus 9/10, Haiku 7/10 with repair, at 5.6× the cost + +Ten questions, scored on compiling; a failure is handed back once with the compiler's own +error, which names the offending path and suggests a real one. + +| model | first try | after one repair | ~cost per query | +| --- | --- | --- | --- | +| Claude Opus 5 | 9/10 | 9/10 | 1.3¢ | +| Claude Haiku 4.5 | 5/10 | 7/10 | **0.23¢** | + +Haiku's failures are inventions of syntax the grammar does not have — `identifiers[*].value`, +`inputs[].components` — rather than misunderstandings of the question. Repair recovers two +of them. That is the failure mode finding 4 would eliminate outright, which is why the +path enum is worth keeping in mind for shallow queries. + +Cost is dominated by the cached prefix: the schema rendering plus the grammar is **15,481 +tokens on Opus, 11,462 on Haiku**, read at a tenth of list price on every call after the +first. Caching mattered more than model choice. + +### 7. The grammar cannot order by a value under a repeated level + +Opus's single failure, unrepairable and shared with Haiku: + +```text +outcomes.products.measurements.percentage.value: max needs a scalar column, not a repeated level +``` + +`Measure` and `Order` require a scalar path, and a yield lives under `outcomes` → +`products` → `measurements`. "The ten highest-yielding reactions" needs a per-reaction +reduction of a repeated path — `max` over each reaction's measurements — and there is no +way to write one. Both models reached for it independently, which is the strongest signal +available that the gap is real rather than a prompting artifact. + +## Conclusions / next steps + +Translation is unconstrained generation checked afterwards, and the design that follows +from that is [beside this entry](assets/nl-search-design.md): one forced tool call over +the recursive grammar with the prefix cached, a coercing parse, one repair turn carrying +the compiler's error, then execution and a short second call that writes the prose. + +Three things to do in order: + +1. **Close the reduction gap** (finding 7). It is a grammar change, it blocks a question + anyone would ask, and the NL layer would otherwise paper over it with a wrong answer. +2. **Build the layer**, with the model as configuration rather than a decision — the + measured 9/10 and 7/10 are ten questions, not an eval set. +3. **Then choose the model on evidence**, including whether path guidance in the cached + prefix closes Haiku's gap. At 5.6× it is worth the measurement. + +## References + +- [`assets/nl-search-design.md`](assets/nl-search-design.md) — the design this entry argues for +- [Structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) — recursion unsupported; limits undocumented +- [anthropic-sdk-python#1185](https://github.com/anthropics/anthropic-sdk-python/issues/1185) — the optional-property state-space report +- [ord-schema#966](https://github.com/open-reaction-database/ord-schema/pull/966) — the split that renamed `agent` to `search` +- [Where the agent search cache can live](../2026-08-15-where-the-search-cache-lives/README.md) — the pivots and index this layer queries through diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md new file mode 100644 index 0000000..3e4dac3 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md @@ -0,0 +1,49 @@ +# Scripts for "What constrains a natural-language layer over the search grammar" + +- **Date:** 2026-08-17 +- **Author:** Steven Kearnes +- **License:** [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/) + +Probes for [the entry beside them](../README.md), and the design they argue for, +[`nl-search-design.md`](nl-search-design.md). + +Every probe but the first makes live Messages API calls. Run them with `ord_schema` +importable and a key in the environment; the interface deployment's key works, and is +where these numbers came from: + +```bash +export ANTHROPIC_API_KEY=$(aws secretsmanager get-secret-value \ + --secret-id ord-interface-anthropic-api-key --query SecretString --output text) +uv run --with anthropic python probe_repair.py claude-haiku-4-5 +``` + +| script | what it measures | finding | +| --- | --- | --- | +| `probe_recursion.py` | what the SDK puts on the wire for a recursive model, via a mock transport — no key needed | 1 | +| `probe_recursion_live.py` | whether `output_config.format` accepts the recursive grammar | 1 | +| `probe_tool_recursion.py` | the same question for tool schemas, strict and not | 1, 5 | +| `probe_shapes.py` | how often a forced tool call parses as written, and after coercion | 5, 6 | +| `probe_stratified.py` | whether an acyclic grammar unlocks structured outputs | 2 | +| `probe_bisect.py` | which construct exhausts the compiled-grammar budget | 2, 3 | +| `probe_strict_tool.py` | whether the strict-tool path has a larger budget | 2 | +| `probe_required.py` | whether requiring every property fits the budget | 3 | +| `probe_flat_union.py` | a flattened predicate with paths enumerated, by depth | 4 | +| `probe_repair.py` | first-try and after-repair accuracy, per model, with cost | 6, 7 | + +Two of these are library rather than probe, and are the reusable part: + +- `stratify.py` turns a recursive JSON Schema into an acyclic one by numbering levels — + a level-*k* definition references level *(k-1)*, and level 0 drops the recursive + branches. Nothing is inlined, so size grows per level rather than per power. +- `require_all.py` makes every property required with an explicit null instead of + optional, rewrites `oneOf` to `anyOf`, and drops keywords a decoder rejects. Optional + properties are what multiply the compiled state machine. + +Both transform only the schema *shown to the model*. The pydantic models are untouched and +still do the validating, which is why dropping `exclusiveMinimum` and the discriminator +costs no correctness. + +`probe_repair.py` takes a model id as its argument and is the one to re-run when the +prompt changes; it prints per-question outcomes, first-try and after-repair tallies, and +the token counts the cost figures come from. Costs in the entry are computed from those +counts at list price, with cached reads at a tenth. diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-design.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-design.md new file mode 100644 index 0000000..4376a1e --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-design.md @@ -0,0 +1,182 @@ +# Natural-language search over the ORD corpus — design + +- **Date:** 2026-08-17 +- **Author:** Steven Kearnes +- **Status:** draft, for review +- **License:** [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/) + +The measurements this argues from are in [the entry beside it](../README.md). Read finding +1–4 before proposing constrained decoding again; it was tried four ways and does not fit. + +## What this builds + +A question in English becomes a validated `Query`, runs against a `Corpus`, and comes back +with the reactions and a sentence saying what they show. One entry point: + +```python +from ord_schema.search import execute, nl + +corpus = execute.Corpus("projections/*/*.parquet", "structures/*/*.parquet") +answer = nl.ask("which reactions use pyridine as a solvent and beat 50% yield?", corpus) + +answer.query # the Query that ran, for display and for a "run it again" button +answer.table # pyarrow.Table, exactly what corpus.search returned +answer.text # "1,204 reactions. Most are Suzuki couplings; yields cluster at 60-75%." +``` + +`ask` is the whole public surface. `translate` and `answer` are separable because the eval +harness needs the first without the second, and a caller that renders its own results +needs the first without the third. + +## What it does not build + +- **Multi-turn.** No clarifying questions, no follow-ups, no session state. A question is + answered or it fails. +- **A second backend.** Corpus only. ord-interface's Postgres path is out of scope, and + the broken `ord_schema.agent.nl_query` import there is a separate cleanup. +- **Constrained decoding.** Measured dead; see the entry. +- **Text search tuning.** Substring predicates remain expressible and remain unoptimized. + +## The gap this depends on + +`Measure` and `Order` require a scalar path, so an aggregate cannot reach a value under a +repeated level: + +```text +outcomes.products.measurements.percentage.value: max needs a scalar column, not a repeated level +``` + +"The ten highest-yielding reactions" is therefore unwritable, and both models tried to +write it anyway. **Task 1 closes this before any NL code exists**, because a layer built +over the gap would answer the question wrongly rather than refuse it. + +The shape: a reduction over a repeated path, usable wherever a scalar is wanted. + +```json +{"order_by": [{"key": {"reduce": "max", "path": "outcomes.products.measurements.percentage.value"}, "descending": true}], + "limit": 10} +``` + +It compiles to a list aggregate over the reaction's own elements — `list_max(list_transform(...))` +over the projection, or `max(...)` over the level's pivot with a correlated subquery — so +it inherits the routing the executor already does per quantifier. `min`, `max`, `avg`, +`sum`, and `count` all make sense; `count` needs no path. + +Filtering on a reduction (`reactions whose best yield beats 50%`) is already expressible as +`exists ... where percentage.value > 50`, and stays that way. This is only about ordering +and aggregating, where there is no quantifier to hang the condition on. + +## How translation works + +```mermaid +flowchart TB + Q["question"] --> T["forced tool call
build_query, non-strict"] + T --> C["coerce: JSON strings to objects"] + C --> V["Query.model_validate"] + V --> K["compile_query"] + K -- "ValidationError or QueryError" --> R{"repaired once?"} + R -- no --> T2["hand back the error
+ its did-you-mean"] + T2 --> C + R -- yes --> F["MalformedQueryError"] + K -- ok --> X["corpus.search"] + X --> S["summarize: counts, columns, a few rows"] + S --> A["second call writes the prose"] +``` + +**The cached prefix** is the system text (instructions plus `schema.describe()`) and the +tool definition, marked `cache_control: {"type": "ephemeral", "ttl": "1h"}`. Measured at +15,481 tokens on Opus and 11,462 on Haiku, read at a tenth of list price on every call +after the first. Two rules keep it cached: nothing volatile goes in the prefix (no +timestamps, no per-request identifiers), and the tool list is built once at import. + +**The coercing parse** walks the tool input and replaces any string that parses as JSON +with the parsed value, because both models return the predicate tree as a JSON string +(finding 5). It runs before pydantic, and it is not a fallback — it is the normal path. + +**One repair turn.** On a validation or compile failure, the assistant's tool call and a +`tool_result` carrying the error text go back with `is_error: true`. The compiler's errors +already name the bad path and suggest a real one, which is what makes this worth doing; +Haiku recovers two of five failures on it. A second failure raises rather than looping — +an unbounded repair loop spends real money discovering the model cannot answer. + +**The answer call** sees a summary, never the table: row count, column names, and up to +five sample rows rendered as text. A result set of 100,000 reactions costs the same +prompt as one of three. + +## Module layout + +| file | holds | +| --- | --- | +| `ord_schema/search/nl.py` | `ask`, `translate`, `answer`, the client, the errors | +| `ord_schema/search/nl_prompt.md` | the system prompt, as markdown rather than a Python string | +| `ord_schema/search/nl_eval.py` | `EvalCase`, `load_cases`, scoring, the report | +| `ord_schema/search/nl_cases.yaml` | the eval set, versioned beside the prompt it grades | + +`nl.py` sits in `search/` because it depends on the grammar, the compiler, and the +executor, and nothing else depends on it. The prompt lives beside it for the reason the +old branch found: a prompt in a markdown file can be edited and diffed as prose. + +Dependencies: a new `nl` extra carrying `anthropic`, since `ord-schema[search]` must stay +installable without it. `ord_schema/dependencies_test.py` gains a profile row, which is +what keeps that promise honest. + +## Errors + +The names come from the `nl-query-backend` branch, because ord-interface already maps them +to status codes and there is no reason to invent new ones: + +| error | when | +| --- | --- | +| `ModelRateLimitedError` | 429 from the API | +| `ModelUnavailableError` | 5xx, timeout, connection failure | +| `MalformedQueryError` | still invalid after the repair turn | + +All three inherit `NLQueryError`. Everything else propagates: a `PairingError` from the +corpus is not the model's fault and should not be dressed up as one. + +## Testing + +**Unit tests take no network.** A fake client returns canned tool calls, which is enough +to pin every behavior that matters: the coercion, the repair turn firing exactly once, the +error taxonomy, the prefix being marked cacheable, and the summary staying bounded as the +table grows. + +**The eval harness is opt-in and costs money.** `nl_eval.py` runs cases against a real +model and reports first-try and after-repair rates, per-question failures, and token cost. +A case states a question and what the answer must satisfy — not a `Query` literal, since +several spellings are right: + +```yaml +- question: reactions using pyridine as the solvent + compiles: true + must_return: [ord-1f2a..., ord-9c7e...] # reactions any correct query returns + must_not_return: [ord-4b81...] # a near-miss the wrong quantifier would include +``` + +Scoring on returned reactions rather than on query shape is what makes the harness able to +say a translation is *wrong* rather than merely *different*. The cases live beside the +prompt so that changing one and not the other shows up in review. + +## What "fast" means here + +A question costs one translation call, one execution, and one answer call. The middle term +is the only one this repository controls, and it is already measured: a structure query +lands in 0.02–1.5 s warm, a quantifier over a pivot in tens of milliseconds. Translation is +seconds and dominated by output tokens; the answer call is shorter. + +So the latency work is not in this layer, and the design should not pretend otherwise. What +this layer owes the corpus is *not making it slow*: no query without a limit, the timeout +passed through to `corpus.search`, and the model never handed enough rows to think it should +summarize them itself. + +## Open questions + +1. **Which model.** Ten questions say Opus 9/10 and Haiku 7/10 at 5.6× less. The eval set + decides, and the interesting variable is whether worked path examples in the cached + prefix close the gap — they are nearly free once cached. +2. **Whether the path enum earns a place.** Enumerating all 431 scalar paths fits a depth-2 + grammar and makes invented paths impossible. If the evals show shallow queries dominate + real traffic, a constrained fast path with an unconstrained fallback becomes tempting — + but it is two systems, and it needs the traffic to justify it. +3. **Where `/ask` lands.** Out of scope here, and the answer probably changes once the + layer exists and ord-interface's dependency on the deleted module is cleaned up. diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_bisect.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_bisect.py new file mode 100644 index 0000000..308697e --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_bisect.py @@ -0,0 +1,60 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Which part of the grammar makes the compiled decoding grammar too large?""" + +import json +import pathlib +import sys + +import anthropic +from anthropic.lib._parse._transform import transform_schema +from pydantic import BaseModel + +sys.path.insert(0, str(pathlib.Path(__file__).parent)) +from stratify import stratify # noqa: E402 + +from ord_schema.search import query # noqa: E402 + + +class Tiny(BaseModel): + name: str + count: int + + +def sized(schema: dict) -> str: + return f"{len(json.dumps(schema)) // 4:,} ~tokens" + + +client = anthropic.Anthropic() +cases = [ + ("a tiny two-field object", transform_schema(Tiny.model_json_schema())), + ("one comparison predicate", transform_schema(query.Comparison.model_json_schema())), + ("one quantifier", transform_schema(query.Quantifier.model_json_schema())), + ("Query, stratified depth 0", transform_schema(stratify(query.Query.model_json_schema(), 0))), + ("Query, stratified depth 1", transform_schema(stratify(query.Query.model_json_schema(), 1))), +] +for label, schema in cases: + try: + client.messages.create( + model="claude-haiku-4-5", + max_tokens=512, + messages=[{"role": "user", "content": "reactions above 350 K"}], + output_config={"format": {"type": "json_schema", "schema": schema}}, + ) + except anthropic.APIStatusError as error: + message = json.loads(error.response.text)["error"]["message"] + print(f" {label:32} {sized(schema):>16} REFUSED: {message[:80]}") + else: + print(f" {label:32} {sized(schema):>16} accepted") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_flat_union.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_flat_union.py new file mode 100644 index 0000000..9171b67 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_flat_union.py @@ -0,0 +1,113 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Can a flattened predicate -- one object per level, path as an enum -- fit the budget? + +The union of eight predicate variants is what multiplies the state machine, so this +collapses them into one object whose `op` says which shape it is. Paths become an enum, +which is the only way the decoder can stop a model inventing `identifiers[*]`. +""" + +import json +import sys + +import anthropic +import pyarrow as pa + +from ord_schema.artifacts import projection + +client = anthropic.Anthropic() + + +def leaf_paths(schema: pa.Schema) -> list[str]: + """Returns every scalar column path in the projection, dotted.""" + found: list[str] = [] + + def walk(field: pa.Field, prefix: str) -> None: + path = f"{prefix}.{field.name}" if prefix else field.name + dtype = field.type + while pa.types.is_list(dtype) or pa.types.is_map(dtype): + dtype = dtype.item_type if pa.types.is_map(dtype) else dtype.value_type + if pa.types.is_struct(dtype): + for child in dtype: + walk(child, path) + else: + found.append(path) + + for field in schema: + walk(field, "") + return found + + +PATHS = leaf_paths(projection.SCHEMA) +print(f"projection leaves: {len(PATHS)}") + + +def predicate(level: int, paths: list[str]) -> dict: + """Returns a flat predicate object whose clauses are one level shallower.""" + node: dict = { + "type": "object", + "additionalProperties": False, + "properties": { + "op": { + "type": "string", + "enum": [ + "and", "or", "not", "exists", "forall", + "eq", "ne", "lt", "le", "gt", "ge", + "contains", "starts_with", "ends_with", + "is_null", "not_null", "substructure", "similarity", + ], + }, + "path": {"anyOf": [{"type": "string", "enum": paths}, {"type": "null"}]}, + "literal": {"anyOf": [{"type": "string"}, {"type": "number"}, + {"type": "boolean"}, {"type": "null"}]}, + "compound": {"anyOf": [{"type": "string"}, {"type": "null"}]}, + "smarts": {"anyOf": [{"type": "string"}, {"type": "null"}]}, + }, + "required": ["op", "path", "literal", "compound", "smarts"], + } + if level > 0: + node["properties"]["clauses"] = { + "anyOf": [{"type": "array", "items": predicate(level - 1, paths)}, + {"type": "null"}] + } + node["required"].append("clauses") + return node + + +for count in (0, 40, 120, len(PATHS)): + paths = PATHS[:count] if count else [""] + for depth in (2, 3, 4): + schema = { + "type": "object", + "additionalProperties": False, + "properties": {"where": predicate(depth, paths)}, + "required": ["where"], + } + if not count: # free-form path, for comparison with the enum forms + schema = json.loads(json.dumps(schema).replace( + '{"type": "string", "enum": [""]}', '{"type": "string"}')) + size = f"{len(json.dumps(schema)) // 4:,} ~tok" + label = f"paths={count or 'free':>4} depth={depth}" + try: + client.messages.create( + model="claude-haiku-4-5", max_tokens=256, + messages=[{"role": "user", "content": "reactions above 350 K"}], + output_config={"format": {"type": "json_schema", "schema": schema}}, + ) + except anthropic.APIStatusError as error: + message = json.loads(error.response.text)["error"]["message"] + print(f" {label} {size:>12} REFUSED: {message[:52]}") + else: + print(f" {label} {size:>12} ACCEPTED") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion.py new file mode 100644 index 0000000..feebe96 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion.py @@ -0,0 +1,103 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""What does the SDK send for a recursive output_format, and does it object first? + +Captures the request body with a mock transport rather than a live call, so the +client-side half of the question is settled without a key: whether the SDK accepts a +self-referencing pydantic model at all, and what JSON Schema it puts on the wire. +""" + +import json + +import anthropic +import httpx + +from ord_schema.search import query + +CANNED = { + "id": "msg_probe", + "type": "message", + "role": "assistant", + "model": "claude-opus-5", + "content": [{"type": "text", "text": "{}"}], + "stop_reason": "end_turn", + "usage": {"input_tokens": 1, "output_tokens": 1}, +} +sent = {} + + +def handler(request: httpx.Request) -> httpx.Response: + sent["body"] = json.loads(request.content) + return httpx.Response(200, json=CANNED) + + +client = anthropic.Anthropic( + api_key="probe-not-a-real-key", + http_client=httpx.Client(transport=httpx.MockTransport(handler)), +) + +print("anthropic SDK:", anthropic.__version__) +for label, call in ( + ( + "messages.parse(output_format=Query)", + lambda: client.messages.parse( + model="claude-opus-5", + max_tokens=1024, + messages=[{"role": "user", "content": "pyridine as solvent"}], + output_format=query.Query, + ), + ), + ( + "messages.create(tools=[strict build_query])", + lambda: client.messages.create( + model="claude-opus-5", + max_tokens=1024, + messages=[{"role": "user", "content": "pyridine as solvent"}], + tools=[ + { + "name": "build_query", + "description": "Build a search query.", + "strict": True, + "input_schema": query.Query.model_json_schema(), + } + ], + tool_choice={"type": "tool", "name": "build_query"}, + ), + ), +): + sent.clear() + print(f"\n=== {label}") + try: + call() + except Exception as error: # noqa: BLE001 -- the probe reports whatever it hits. + print(f" raised {type(error).__name__}: {str(error)[:200]}") + body = sent.get("body") + if body is None: + print(" nothing reached the wire") + continue + schema = None + if "output_config" in body: + schema = body["output_config"].get("format", {}).get("schema") + print(" output_config.format keys:", sorted(body["output_config"]["format"])) + elif "tools" in body: + schema = body["tools"][0].get("input_schema") + print(" tool keys:", sorted(body["tools"][0])) + if schema is None: + print(" no schema on the wire") + continue + text = json.dumps(schema) + print(f" schema on the wire: {len(text)} chars, $defs={len(schema.get('$defs', {}))}") + print(f" contains $ref: {'$ref' in text}, refs={text.count('#/$defs/')}") + print(f" additionalProperties present: {text.count('additionalProperties')}") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion_live.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion_live.py new file mode 100644 index 0000000..419290c --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_recursion_live.py @@ -0,0 +1,57 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Does the API accept a recursive json_schema output format? Needs a real key. + + ANTHROPIC_API_KEY=sk-... uv run --with anthropic python probe_recursion_live.py + +Prints the validated Query the model produced, or the error the server returned. +""" + +import json + +import anthropic + +from ord_schema.search import query, schema + +SYSTEM = ( + "You translate chemistry questions into ORD search queries. The corpus schema, " + "as an indented type tree in DuckDB's types:\n\n" + schema.describe() +) + +client = anthropic.Anthropic() +try: + response = client.messages.parse( + model="claude-opus-5", + max_tokens=2048, + system=[{"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}}], + messages=[ + { + "role": "user", + "content": "reactions using pyridine as the solvent with a yield above 50%", + } + ], + output_format=query.Query, + ) +except anthropic.APIStatusError as error: + print(f"server refused it: {error.status_code} {str(error)[:400]}") +else: + print("accepted. parsed_output:") + print(json.dumps(response.parsed_output.model_dump(exclude_none=True), indent=2)) + usage = response.usage + print( + f"\ninput={usage.input_tokens} " + f"cache_write={usage.cache_creation_input_tokens} " + f"cache_read={usage.cache_read_input_tokens} output={usage.output_tokens}" + ) diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_repair.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_repair.py new file mode 100644 index 0000000..2220dd4 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_repair.py @@ -0,0 +1,130 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Does one repair turn carry a cheap model to the accuracy of an expensive one? + +The compiler's errors name the offending path and suggest a real one, so a failed +translation is handed straight back. Scores first-try and after-repair separately. +""" + +import json +import sys + +import anthropic + +from ord_schema.search import query, schema + +QUESTIONS = [ + "reactions using pyridine as the solvent", + "reactions using pyridine as the solvent with a yield above 50%", + "reactions run above 350 K", + "reactions where every input component is a liquid", + "average yield by reaction temperature, for reactions with a boronic acid", + "reactions that make aspirin", + "the ten highest-yielding Suzuki couplings", + "reactions with no solvent at all", + "how many reactions use palladium catalysts", + "reactions stirred for more than an hour at above 100 C", +] +SYSTEM = [ + { + "type": "text", + "text": ( + "You translate chemistry questions into ORD search queries by calling " + "build_query. The corpus schema, as an indented type tree in DuckDB's " + "types:\n\n" + schema.describe() + ), + "cache_control": {"type": "ephemeral"}, + } +] +TOOL = { + "name": "build_query", + "description": "Build an ORD search query from the user's question.", + "input_schema": query.Query.model_json_schema(), +} +MODEL = sys.argv[1] if len(sys.argv) > 1 else "claude-haiku-4-5" + + +def coerce(value): + """Returns the input with any JSON-encoded string values parsed back to objects.""" + if isinstance(value, str): + try: + return coerce(json.loads(value)) + except (json.JSONDecodeError, TypeError): + return value + if isinstance(value, dict): + return {key: coerce(item) for key, item in value.items()} + if isinstance(value, list): + return [coerce(item) for item in value] + return value + + +def check(raw) -> str | None: + """Returns the error a translation fails with, or None if it compiles.""" + try: + query.compile_query(query.Query.model_validate(coerce(raw))) + except Exception as error: # noqa: BLE001 -- whatever it fails with is the message. + return str(error) + return None + + +client = anthropic.Anthropic() +first_try = repaired = 0 +cost = {"cache_read": 0, "input": 0, "output": 0} +for question in QUESTIONS: + messages: list = [{"role": "user", "content": question}] + response = client.messages.create( + model=MODEL, max_tokens=2048, system=SYSTEM, messages=messages, + tools=[TOOL], tool_choice={"type": "tool", "name": "build_query"}, + ) + cost["cache_read"] += response.usage.cache_read_input_tokens or 0 + cost["input"] += response.usage.input_tokens + cost["output"] += response.usage.output_tokens + block = next(b for b in response.content if b.type == "tool_use") + error = check(block.input) + if error is None: + first_try += 1 + repaired += 1 + print(f" {question[:48]:50} ok") + continue + messages += [ + {"role": "assistant", "content": response.content}, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": block.id, + "is_error": True, + "content": f"That query was rejected: {error}. Call build_query again with it fixed.", + } + ], + }, + ] + retry = client.messages.create( + model=MODEL, max_tokens=2048, system=SYSTEM, messages=messages, + tools=[TOOL], tool_choice={"type": "tool", "name": "build_query"}, + ) + cost["cache_read"] += retry.usage.cache_read_input_tokens or 0 + cost["input"] += retry.usage.input_tokens + cost["output"] += retry.usage.output_tokens + retry_block = next(b for b in retry.content if b.type == "tool_use") + second = check(retry_block.input) + if second is None: + repaired += 1 + print(f" {question[:48]:50} repaired ({error.split(';')[0][:44]})") + else: + print(f" {question[:48]:50} STILL FAILS: {second[:60]}") +print(f"\n{MODEL}: first try {first_try}/{len(QUESTIONS)}, after repair {repaired}/{len(QUESTIONS)}") +print(f"tokens: {cost}") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_required.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_required.py new file mode 100644 index 0000000..b580fb8 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_required.py @@ -0,0 +1,46 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Does requiring every property bring the grammar under the compiler's budget?""" + +import copy +import json +import pathlib +import sys + +import anthropic + +sys.path.insert(0, str(pathlib.Path(__file__).parent)) +from require_all import require_all # noqa: E402 +from stratify import stratify # noqa: E402 + +from ord_schema.search import query # noqa: E402 + +client = anthropic.Anthropic() +base = query.Query.model_json_schema() +for depth in range(0, 6): + schema = require_all(stratify(copy.deepcopy(base), depth), strip_descriptions=True) + size = f"{len(json.dumps(schema)) // 4:,} ~tokens" + try: + client.messages.create( + model="claude-haiku-4-5", + max_tokens=512, + messages=[{"role": "user", "content": "reactions above 350 K"}], + output_config={"format": {"type": "json_schema", "schema": schema}}, + ) + except anthropic.APIStatusError as error: + message = json.loads(error.response.text)["error"]["message"] + print(f" depth {depth} {size:>14} REFUSED: {message[:78]}") + else: + print(f" depth {depth} {size:>14} ACCEPTED") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_shapes.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_shapes.py new file mode 100644 index 0000000..387845f --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_shapes.py @@ -0,0 +1,110 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""How often does a forced tool call over the recursive grammar parse as written? + +Runs a handful of questions through two models and scores three ways: as returned, +after coercing JSON-encoded strings back to objects, and whether the compiler accepts +the result. The gap between the first two is what a strict schema would have bought. +""" + +import json + +import anthropic + +from ord_schema.search import query, schema + +QUESTIONS = [ + "reactions using pyridine as the solvent", + "reactions using pyridine as the solvent with a yield above 50%", + "reactions run above 350 K", + "reactions where every input component is a liquid", + "average yield by reaction temperature, for reactions with a boronic acid", +] +SYSTEM = [ + { + "type": "text", + "text": ( + "You translate chemistry questions into ORD search queries. Emit the query " + "by calling build_query. The corpus schema, as an indented type tree in " + "DuckDB's types:\n\n" + schema.describe() + ), + "cache_control": {"type": "ephemeral"}, + } +] +TOOL = { + "name": "build_query", + "description": "Build an ORD search query from the user's question.", + "input_schema": query.Query.model_json_schema(), +} + + +def coerce(value): + """Returns the input with any JSON-encoded string values parsed back to objects.""" + if isinstance(value, str): + try: + return coerce(json.loads(value)) + except (json.JSONDecodeError, TypeError): + return value + if isinstance(value, dict): + return {key: coerce(item) for key, item in value.items()} + if isinstance(value, list): + return [coerce(item) for item in value] + return value + + +client = anthropic.Anthropic() +for model in ("claude-opus-5", "claude-haiku-4-5"): + print(f"\n=== {model}") + tallies = {"as written": 0, "coerced": 0, "compiles": 0} + cost = {"cache_write": 0, "cache_read": 0, "input": 0, "output": 0} + for question in QUESTIONS: + response = client.messages.create( + model=model, + max_tokens=2048, + system=SYSTEM, + messages=[{"role": "user", "content": question}], + tools=[TOOL], + tool_choice={"type": "tool", "name": "build_query"}, + ) + usage = response.usage + cost["cache_write"] += usage.cache_creation_input_tokens or 0 + cost["cache_read"] += usage.cache_read_input_tokens or 0 + cost["input"] += usage.input_tokens + cost["output"] += usage.output_tokens + raw = next(b for b in response.content if b.type == "tool_use").input + note = [] + try: + query.Query.model_validate(raw) + except Exception: # noqa: BLE001 -- scoring, not handling. + note.append("as-written FAILED") + else: + tallies["as written"] += 1 + note.append("as-written ok") + try: + parsed = query.Query.model_validate(coerce(raw)) + except Exception as error: # noqa: BLE001 + note.append(f"coerced FAILED: {str(error)[:60]}") + else: + tallies["coerced"] += 1 + try: + query.compile_query(parsed) + except Exception as error: # noqa: BLE001 + note.append(f"compile FAILED: {str(error)[:80]}") + else: + tallies["compiles"] += 1 + note.append("compiles") + print(f" {question[:52]:54} {' | '.join(note)}") + print(f" tallies (of {len(QUESTIONS)}): {tallies}") + print(f" tokens: {cost}") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_stratified.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_stratified.py new file mode 100644 index 0000000..1617419 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_stratified.py @@ -0,0 +1,93 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Does a stratified grammar unlock structured outputs, and can a cheap model hit it? + +Same five questions as the tool-call probe, but the model is handed an acyclic schema +and constrained by output_config.format rather than asked to call a tool. +""" + +import json +import pathlib +import sys + +import anthropic +from anthropic.lib._parse._transform import transform_schema + +sys.path.insert(0, str(pathlib.Path(__file__).parent)) +from stratify import stratify # noqa: E402 + +from ord_schema.search import query, schema # noqa: E402 + +QUESTIONS = [ + "reactions using pyridine as the solvent", + "reactions using pyridine as the solvent with a yield above 50%", + "reactions run above 350 K", + "reactions where every input component is a liquid", + "average yield by reaction temperature, for reactions with a boronic acid", +] +SYSTEM = [ + { + "type": "text", + "text": ( + "You translate chemistry questions into ORD search queries. The corpus " + "schema, as an indented type tree in DuckDB's types:\n\n" + schema.describe() + ), + "cache_control": {"type": "ephemeral"}, + } +] +DEPTH = int(sys.argv[1]) if len(sys.argv) > 1 else 4 +MODEL = sys.argv[2] if len(sys.argv) > 2 else "claude-haiku-4-5" +STRATIFIED = transform_schema(stratify(query.Query.model_json_schema(), DEPTH)) + +client = anthropic.Anthropic() +print(f"=== {MODEL}, stratified depth {DEPTH} " + f"({len(json.dumps(STRATIFIED))//4:,} ~tokens)") +tallies = {"parses": 0, "compiles": 0} +cost = {"cache_write": 0, "cache_read": 0, "input": 0, "output": 0} +for question in QUESTIONS: + try: + response = client.messages.create( + model=MODEL, + max_tokens=2048, + system=SYSTEM, + messages=[{"role": "user", "content": question}], + output_config={"format": {"type": "json_schema", "schema": STRATIFIED}}, + ) + except anthropic.APIStatusError as error: + print(f" refused: {error.status_code} {str(error)[:300]}") + break + usage = response.usage + cost["cache_write"] += usage.cache_creation_input_tokens or 0 + cost["cache_read"] += usage.cache_read_input_tokens or 0 + cost["input"] += usage.input_tokens + cost["output"] += usage.output_tokens + text = next(b.text for b in response.content if b.type == "text") + note = [] + try: + parsed = query.Query.model_validate(json.loads(text)) + except Exception as error: # noqa: BLE001 -- scoring, not handling. + note.append(f"parse FAILED: {str(error)[:90]}") + else: + tallies["parses"] += 1 + try: + query.compile_query(parsed) + except Exception as error: # noqa: BLE001 + note.append(f"compile FAILED: {str(error)[:90]}") + else: + tallies["compiles"] += 1 + note.append("compiles") + print(f" {question[:52]:54} {' | '.join(note)}") +print(f" tallies (of {len(QUESTIONS)}): {tallies}") +print(f" tokens: {cost}") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_strict_tool.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_strict_tool.py new file mode 100644 index 0000000..eb52917 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_strict_tool.py @@ -0,0 +1,52 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Does the strict-tool path have a bigger grammar budget than output_config.format?""" + +import json +import pathlib +import sys + +import anthropic +from anthropic.lib._parse._transform import transform_schema + +sys.path.insert(0, str(pathlib.Path(__file__).parent)) +from stratify import stratify # noqa: E402 + +from ord_schema.search import query # noqa: E402 + +client = anthropic.Anthropic() +base = query.Query.model_json_schema() +for depth in (0, 1, 2, 4): + schema = transform_schema(stratify(base, depth)) + tool = { + "name": "build_query", + "description": "Build an ORD search query.", + "strict": True, + "input_schema": schema, + } + size = f"{len(json.dumps(schema)) // 4:,} ~tokens" + try: + client.messages.create( + model="claude-haiku-4-5", + max_tokens=512, + messages=[{"role": "user", "content": "reactions above 350 K"}], + tools=[tool], + tool_choice={"type": "tool", "name": "build_query"}, + ) + except anthropic.APIStatusError as error: + message = json.loads(error.response.text)["error"]["message"] + print(f" strict tool, depth {depth} {size:>16} REFUSED: {message[:70]}") + else: + print(f" strict tool, depth {depth} {size:>16} accepted") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_tool_recursion.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_tool_recursion.py new file mode 100644 index 0000000..e39180b --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/probe_tool_recursion.py @@ -0,0 +1,80 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Do tool input schemas accept the recursion that output_config.format refuses? + +Three shapes, cheapest question that still needs a nested predicate: a forced tool call +over the grammar as pydantic writes it, the same with strict on and the SDK's own +closing transform applied, and -- for reference -- what the model actually produces. +""" + +import json + +import anthropic +from anthropic.lib._parse._transform import transform_schema + +from ord_schema.search import query, schema + +QUESTION = "reactions using pyridine as the solvent with a yield above 50%" +SYSTEM = [ + { + "type": "text", + "text": ( + "You translate chemistry questions into ORD search queries. The corpus " + "schema, as an indented type tree in DuckDB's types:\n\n" + schema.describe() + ), + "cache_control": {"type": "ephemeral"}, + } +] +RAW = query.Query.model_json_schema() + +client = anthropic.Anthropic() +for label, input_schema, strict in ( + ("tool, as pydantic writes it", RAW, False), + ("tool, strict + closed", transform_schema(json.loads(json.dumps(RAW))), True), +): + tool: dict = { + "name": "build_query", + "description": "Build an ORD search query from the user's question.", + "input_schema": input_schema, + } + if strict: + tool["strict"] = True + print(f"\n=== {label}") + try: + response = client.messages.create( + model="claude-opus-5", + max_tokens=2048, + system=SYSTEM, + messages=[{"role": "user", "content": QUESTION}], + tools=[tool], + tool_choice={"type": "tool", "name": "build_query"}, + ) + except anthropic.APIStatusError as error: + print(f" refused: {error.status_code} {str(error)[:220]}") + continue + block = next(b for b in response.content if b.type == "tool_use") + print(" accepted. tool_use.input:") + print(json.dumps(block.input, indent=2)[:900]) + try: + parsed = query.Query.model_validate(block.input) + except Exception as error: # noqa: BLE001 -- the probe reports whatever it hits. + print(f" pydantic REJECTED it: {type(error).__name__}: {str(error)[:200]}") + else: + print(f" pydantic accepted it: {parsed.where.op}") + usage = response.usage + print( + f" input={usage.input_tokens} cache_write={usage.cache_creation_input_tokens} " + f"cache_read={usage.cache_read_input_tokens} output={usage.output_tokens}" + ) diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/require_all.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/require_all.py new file mode 100644 index 0000000..83c9b6c --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/require_all.py @@ -0,0 +1,105 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Makes every property required, expressing optionality as an explicit null instead. + +Constrained decoding compiles a schema into a state machine, and an optional property +doubles it: the machine has to accept the object with and without that key. Requiring +every key and letting the value be null costs one token per absent field and leaves the +machine linear in the number of properties. +""" + +import copy + +# Keywords a decoder rejects or ignores. Dropping them costs nothing: the response is +# validated by the pydantic models afterwards, which enforce all of them anyway. +_UNSUPPORTED = frozenset( + { + "minimum", + "maximum", + "exclusiveMinimum", + "exclusiveMaximum", + "multipleOf", + "minLength", + "maxLength", + "pattern", + "maxItems", + "uniqueItems", + "default", + } +) + + +def require_all(node, strip_descriptions: bool = False): + """Returns node with every property required and nullable where it was optional.""" + if isinstance(node, list): + return [require_all(item, strip_descriptions) for item in node] + if not isinstance(node, dict): + return node + out = {} + for key, value in node.items(): + if strip_descriptions and key in ("description", "title", "examples"): + continue + # A decoder takes anyOf but not oneOf, and the discriminator that rides with it + # is pydantic's business rather than the model's: validation still happens here. + if key == "discriminator" or key in _UNSUPPORTED: + continue + out["anyOf" if key == "oneOf" else key] = require_all(value, strip_descriptions) + properties = out.get("properties") + if isinstance(properties, dict) and properties: + required = set(out.get("required", [])) + for name, subschema in properties.items(): + if name in required: + continue + branches = subschema.get("anyOf") if isinstance(subschema, dict) else None + if branches and any(b.get("type") == "null" for b in branches): + continue + properties[name] = {"anyOf": [subschema, {"type": "null"}]} + out["required"] = sorted(properties) + out["additionalProperties"] = False + return out + + +def count_optional(node, seen=None) -> int: + """Returns how many properties a schema leaves optional.""" + total = 0 + if isinstance(node, dict): + properties = node.get("properties") + if isinstance(properties, dict): + total += len(set(properties) - set(node.get("required", []))) + for value in node.values(): + total += count_optional(value) + elif isinstance(node, list): + for item in node: + total += count_optional(item) + return total + + +if __name__ == "__main__": + import json + import sys + + sys.path.insert(0, str(pathlib.Path(__file__).parent)) + from stratify import stratify + + from ord_schema.search import query + + base = query.Query.model_json_schema() + print(f"{'depth':>6} {'optional before':>16} {'after':>6} {'~tokens':>9}") + for depth in range(0, 5): + built = stratify(copy.deepcopy(base), depth) + before = count_optional(built) + required = require_all(built, strip_descriptions=True) + text = json.dumps(required) + print(f"{depth:>6} {before:>16} {count_optional(required):>6} {len(text)//4:>9,}") diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/stratify.py b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/stratify.py new file mode 100644 index 0000000..2ff2ce5 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/stratify.py @@ -0,0 +1,109 @@ +# Copyright 2026 Open Reaction Database Project Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Turns the recursive grammar into an acyclic one by stratifying it into levels. + +The API refuses a schema whose definitions reference each other in a cycle, which is +what a predicate tree is. Stratifying keeps every definition but numbers it: a level-k +predicate's clauses are level-(k-1) predicates, and level 0 holds only the leaves. The +refs stay refs -- nothing is inlined, so size grows by a level rather than by a power -- +and the result validates into the same recursive pydantic models. +""" + +import copy +import json + + +def _referenced(node, found): + """Collects the definition names ``node`` references.""" + if isinstance(node, dict): + ref = node.get("$ref") + if ref: + found.add(ref.split("/")[-1]) + for value in node.values(): + _referenced(value, found) + elif isinstance(node, list): + for item in node: + _referenced(item, found) + return found + + +def _retarget(node, level, levelled): + """Returns node with refs pointing at the next level down.""" + if isinstance(node, dict): + ref = node.get("$ref") + if ref: + name = ref.split("/")[-1] + if name not in levelled: + return {"$ref": f"#/$defs/{name}"} + if level <= 0: + return None + return {"$ref": f"#/$defs/{name}_L{level - 1}"} + out = {} + for key, value in node.items(): + result = _retarget(value, level, levelled) + if result is None: + if key in ("anyOf", "items", "properties"): + return None + continue + if isinstance(result, list) and not result: + return None + out[key] = result + if "anyOf" in out and not out["anyOf"]: + return None + return out + if isinstance(node, list): + kept = [_retarget(item, level, levelled) for item in node] + return [item for item in kept if item is not None] + return node + + +def stratify(schema: dict, depth: int) -> dict: + """Returns an acyclic schema whose predicates nest at most ``depth`` levels. + + Args: + schema: A JSON Schema with a recursive ``$defs`` section. + depth: How many levels of nesting to allow; level 0 holds only leaves. + + Returns: + A schema with no reference cycle, its definitions suffixed by level. + """ + schema = copy.deepcopy(schema) + defs = schema.pop("$defs", {}) + # A definition that reaches another definition is one a level has to renumber; + # leaves reference nothing, so one copy of each serves every level. + levelled = {name for name, node in defs.items() if _referenced(node, set())} + out: dict = {} + for name, node in defs.items(): + if name not in levelled: + out[name] = node + continue + for level in range(depth + 1): + built = _retarget(copy.deepcopy(node), level, levelled) + if built is not None: + out[f"{name}_L{level}"] = built + root = _retarget(schema, depth, levelled) + root["$defs"] = out + return root + + +if __name__ == "__main__": + from ord_schema.search import query + + base = query.Query.model_json_schema() + print(f"{'depth':>6} {'chars':>9} {'~tokens':>8} {'defs':>6}") + for depth in range(1, 7): + built = stratify(base, depth) + text = json.dumps(built) + print(f"{depth:>6} {len(text):>9,} {len(text)//4:>8,} {len(built['$defs']):>6}") From 667794db9b5097f9bd83e4bf4e641185704c43ff Mon Sep 17 00:00:00 2001 From: Steven Kearnes Date: Mon, 17 Aug 2026 21:46:01 -0400 Subject: [PATCH 2/3] Add the implementation plan beside the design Seven tasks: the reduction that closes the grammar gap first, then the module and its extra, translation with the coercing parse, the repair turn pinned to one attempt, the summary and prose answer, the round trip, and an eval harness that scores on the reactions a query returns rather than on how the query is spelled. Co-Authored-By: Claude Opus 5 (1M context) --- .../README.md | 1 + .../assets/README.md | 5 +- .../assets/nl-search-plan.md | 1073 +++++++++++++++++ 3 files changed, 1077 insertions(+), 2 deletions(-) create mode 100644 entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-plan.md diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md index 04d575f..7a218a8 100644 --- a/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md @@ -194,6 +194,7 @@ Three things to do in order: ## References - [`assets/nl-search-design.md`](assets/nl-search-design.md) — the design this entry argues for +- [`assets/nl-search-plan.md`](assets/nl-search-plan.md) — the task-by-task plan built from it - [Structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) — recursion unsupported; limits undocumented - [anthropic-sdk-python#1185](https://github.com/anthropics/anthropic-sdk-python/issues/1185) — the optional-property state-space report - [ord-schema#966](https://github.com/open-reaction-database/ord-schema/pull/966) — the split that renamed `agent` to `search` diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md index 3e4dac3..6bcdc9a 100644 --- a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/README.md @@ -4,8 +4,9 @@ - **Author:** Steven Kearnes - **License:** [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/) -Probes for [the entry beside them](../README.md), and the design they argue for, -[`nl-search-design.md`](nl-search-design.md). +Probes for [the entry beside them](../README.md), the design they argue for +([`nl-search-design.md`](nl-search-design.md)), and the plan that implements it +([`nl-search-plan.md`](nl-search-plan.md)). Every probe but the first makes live Messages API calls. Run them with `ord_schema` importable and a key in the environment; the interface deployment's key works, and is diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-plan.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-plan.md new file mode 100644 index 0000000..c457af3 --- /dev/null +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/assets/nl-search-plan.md @@ -0,0 +1,1073 @@ +# Natural-language search implementation plan + +- **Date:** 2026-08-17 +- **Author:** Steven Kearnes +- **Status:** draft +- **License:** [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A chemist's question becomes a validated `Query`, runs against a `Corpus`, and comes back with the reactions and a sentence about them. + +**Architecture:** One forced tool call over the recursive grammar with the ~15K-token prefix cached; the response is coerced (both models return the predicate tree as a JSON string), validated by the existing pydantic models, and compiled. A failure is handed back once with the compiler's own error, which names the bad path and suggests a real one. Execution goes through `Corpus.search`; a second short call sees a summary of the result — never the table — and writes the prose. + +**Tech Stack:** Python 3.11+, pydantic 2, `anthropic` SDK, DuckDB via `ord_schema.search.execute`, pytest. + +**Spec:** [`nl-search-design.md`](nl-search-design.md), and the measurements it argues from in [the entry](../README.md). + +## Global Constraints + +- **Model is configuration, never a hard-coded decision.** `DEFAULT_MODEL = "claude-haiku-4-5"`; every entry point takes `model`. +- **The cached prefix must stay byte-stable.** No timestamps, no per-request identifiers, no dict iteration order in the system text or the tool definition. Cache reads are ~90% of the per-query cost. +- **`ord-schema[search]` must remain installable without `anthropic`.** The new dependency goes in an `nl` extra, and `ord_schema/dependencies_test.py` gains a profile row that proves it. +- **Repair runs at most once.** A second failure raises `MalformedQueryError`. +- **No lazy imports.** Every import at module top, per the repository's Python style. +- **Docstrings are Google style**, summary line on one physical line, `Args:`/`Returns:`/`Raises:` where non-empty. +- **Ruff formats at line length 88**; `uv run ruff format` and `uv run ruff check` must pass before every commit. +- **Tests run offline.** Only `nl_eval.py` calls the API, and nothing in the suite invokes it. + +--- + +## File structure + +| file | responsibility | +| --- | --- | +| `ord_schema/search/query.py` (modify) | `Reduction`, and the reduction arms of `Order.key` and `Measure.path` | +| `ord_schema/search/nl.py` (create) | errors, client, `translate`, `summarize`, `answer`, `ask` | +| `ord_schema/search/nl_prompt.md` (create) | the system prompt, as prose | +| `ord_schema/search/nl_test.py` (create) | every behavior above, against a stub client | +| `ord_schema/search/nl_eval.py` (create) | `EvalCase`, `load_cases`, `run_case`, `report` | +| `ord_schema/search/nl_cases.yaml` (create) | the eval set | +| `pyproject.toml` (modify) | the `nl` extra | +| `ord_schema/dependencies_test.py` (modify) | the `nl` profile | + +--- + +### Task 1: A reduction over a repeated path + +The grammar cannot order by a value under a repeated level, so "the ten highest-yielding reactions" is unwritable. `resolve()` already returns a *list* expression for a repeated path, so a reduction is one DuckDB list aggregate around it. + +**Files:** + +- Modify: `ord_schema/search/query.py` (`Order`, `Measure`, `compile_query`) +- Test: `ord_schema/search/query_test.py` + +**Interfaces:** + +- Consumes: `resolve(path, schema=...) -> _Resolved(expression, repeated, dtype)`, `QueryError`. +- Produces: `Reduction` with fields `reduce: Literal["min","max","avg","sum","count"]` and `path: str`; `Order.key: str | Reduction`; `Measure.path: str | Reduction | None`. + +- [ ] **Step 1: Write the failing test** + +```python +def test_ordering_by_a_reduction_over_a_repeated_path(): + compiled = query.compile_query( + query.Query.model_validate( + { + "order_by": [ + { + "key": { + "reduce": "max", + "path": "outcomes.products.measurements.percentage.value", + }, + "descending": True, + } + ], + "limit": 10, + } + ) + ) + assert "list_max(" in compiled.sql + assert compiled.sql.endswith("DESC LIMIT 10") + + +def test_a_reduction_over_a_scalar_path_is_refused(): + # A scalar needs no reducing, and accepting one would give two spellings for the + # same query -- one of which silently wraps a value in a single-element list. + with pytest.raises(query.QueryError, match="already scalar"): + query.compile_query( + query.Query.model_validate( + { + "order_by": [ + { + "key": { + "reduce": "max", + "path": "conditions.temperature.setpoint_kelvin", + } + } + ] + } + ) + ) +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest ord_schema/search/query_test.py -k reduction -v` +Expected: FAIL — `Order.key` rejects a dict, so pydantic raises a `ValidationError`. + +- [ ] **Step 3: Add the model and the compiler support** + +In `ord_schema/search/query.py`, above `class Order`: + +```python +# DuckDB's list aggregates, which ignore nulls; count is the number of values that are +# actually there, so it filters rather than taking len() of a list holding nulls. +_REDUCERS = { + "min": "list_min({expression})", + "max": "list_max({expression})", + "avg": "list_avg({expression})", + "sum": "list_sum({expression})", + "count": "len(list_filter({expression}, value -> value IS NOT NULL))", +} + + +class Reduction(BaseModel): + """One value per reaction, reduced from a path that crosses a repeated level. + + An ordering key and an aggregate's argument both have to be scalar, which leaves + "the highest-yielding reactions" unwritable: a yield lives under outcomes, + products, and measurements, so the path resolves to a list. This reduces that list + to the one number the reaction is judged by. + + Attributes: + reduce: How to reduce the list to a value. + path: A dotted path that crosses at least one repeated level. + """ + + reduce: Literal["min", "max", "avg", "sum", "count"] + path: str +``` + +And the resolver, beside `_scalar`: + +```python +def _reduced(reduction: Reduction, schema: pa.Schema) -> str: + """Returns the expression reducing a repeated path to one value per reaction. + + Args: + reduction: What to reduce, and how. + schema: Schema the path is resolved against. + + Returns: + A DuckDB expression yielding one scalar per reaction, NULL where the reaction + holds no elements at all. + + Raises: + QueryError: If the path does not cross a repeated level, since a scalar path + needs no reduction and accepting one would give a query two spellings. + """ + resolved = resolve(reduction.path, schema=schema) + if not resolved.repeated: + raise QueryError( + f"{reduction.path}: {reduction.reduce} reduces a repeated level, and this " + f"path is already scalar; order by the path itself" + ) + return _REDUCERS[reduction.reduce].format(expression=resolved.expression) +``` + +Widen the two fields: + +```python +class Order(BaseModel): + """How to sort the result.""" + + key: str | Reduction + descending: bool = False +``` + +```python + fn: Literal["count", "count_distinct", "sum", "avg", "min", "max"] + path: str | Reduction | None = None + name: str +``` + +- [ ] **Step 4: Teach `compile_query` both places** + +The measure argument, replacing `argument = _scalar(measure.path, schema, measure.fn)`: + +```python + elif isinstance(measure.path, Reduction): + argument = _reduced(measure.path, schema) + else: + argument = _scalar(measure.path, schema, measure.fn) +``` + +The ordering key, at the top of the `for order in query.order_by:` body: + +```python + if isinstance(order.key, Reduction): + key = _reduced(order.key, schema) + elif orderable is None: +``` + +An aggregated query still orders by a measure name or a `group_by` path; a reduction is a per-reaction value and there is no such row after grouping. Leave that arm as it is — a `Reduction` reaching it is caught by the `isinstance` above only for the ungrouped case, so add the guard: + +```python + if isinstance(order.key, Reduction) and orderable is not None: + raise QueryError( + "an aggregated query orders by a measure name or a group_by path; " + "reduce inside a measure instead" + ) +``` + +- [ ] **Step 5: Run the tests** + +Run: `uv run pytest ord_schema/search/query_test.py -v` +Expected: PASS, including the two new tests. + +- [ ] **Step 6: Check the reduction against real data** + +Run: + +```bash +uv run python -c " +from ord_schema.search import execute, query +corpus = execute.Corpus('~/ord/projections/**/*.parquet', '~/ord/structures/**/*.parquet', + require_current=False, resolver={}.__getitem__) +table = corpus.search(query.Query.model_validate({ + 'order_by': [{'key': {'reduce': 'max', 'path': 'outcomes.products.measurements.percentage.value'}, + 'descending': True}], + 'limit': 10})) +print(table.num_rows) +" +``` + +Expected: 10 rows. This is the question both models tried to write and could not. + +- [ ] **Step 7: Document it** + +Add a row to the worked-example table in `ord_schema/search/README.md`: + +```markdown +| the ten highest-yielding reactions | `order_by` a `reduce` over `outcomes.products.measurements` | no quantifier: a list aggregate over the projection | +``` + +- [ ] **Step 8: Commit** + +```bash +git add ord_schema/search/query.py ord_schema/search/query_test.py ord_schema/search/README.md +git commit -m "Let a query order by a value under a repeated level" +``` + +--- + +### Task 2: The package, its dependency, and its errors + +**Files:** + +- Create: `ord_schema/search/nl.py`, `ord_schema/search/nl_test.py` +- Modify: `pyproject.toml`, `ord_schema/dependencies_test.py` + +**Interfaces:** + +- Produces: `NLQueryError`, `ModelUnavailableError`, `ModelRateLimitedError`, `MalformedQueryError`, `DEFAULT_MODEL = "claude-haiku-4-5"`, `get_client() -> anthropic.Anthropic`. + +- [ ] **Step 1: Write the failing test** + +```python +def test_the_errors_share_one_base(): + # ord-interface maps these onto status codes, so a caller can catch the base and + # still tell a rate limit from a query it could not build. + assert issubclass(nl.ModelRateLimitedError, nl.NLQueryError) + assert issubclass(nl.ModelUnavailableError, nl.NLQueryError) + assert issubclass(nl.MalformedQueryError, nl.NLQueryError) +``` + +- [ ] **Step 2: Run it** + +Run: `uv run pytest ord_schema/search/nl_test.py -v` +Expected: FAIL — `ModuleNotFoundError: ord_schema.search.nl`. + +- [ ] **Step 3: Write the module head** + +```python +"""Natural-language questions, translated into the search grammar and answered. + +A model cannot be constrained to emit a valid Query: the grammar is recursive, and both +constrained-decoding paths refuse it -- the measurements are in the ord-logbook entry +"What constrains a natural-language layer over the search grammar". So translation is +generation checked afterwards: the response is coerced, validated by the same models the +compiler uses, and handed back once with the compiler's error when it does not compile. +""" + +import dataclasses +import json +from importlib import resources +from typing import Any + +import anthropic +import pyarrow as pa + +from ord_schema.logging import get_logger +from ord_schema.search import execute, query, schema + +logger = get_logger(__name__) + +DEFAULT_MODEL = "claude-haiku-4-5" +MAX_TOKENS = 2048 + + +class NLQueryError(Exception): + """A question could not be answered.""" + + +class ModelUnavailableError(NLQueryError): + """The model could not be reached.""" + + +class ModelRateLimitedError(NLQueryError): + """The model refused the request for rate reasons.""" + + +class MalformedQueryError(NLQueryError): + """The model's query did not validate, and the repair attempt did not either.""" + + +def get_client() -> anthropic.Anthropic: + """Returns a client reading its credentials from the environment.""" + return anthropic.Anthropic() +``` + +- [ ] **Step 4: Run it** + +Run: `uv run pytest ord_schema/search/nl_test.py -v` +Expected: PASS. + +- [ ] **Step 5: Declare the dependency** + +In `pyproject.toml`, after the `search` extra: + +```toml +# Optional: ord_schema.search.nl turns a question into a Query by asking a model, so it +# needs an API client where the rest of the search subpackage needs none. +nl = [ + "anthropic>=0.120", + "ord-schema[search]", +] +``` + +In `ord_schema/dependencies_test.py`, add to both dictionaries: + +```python + "nl": ("ord_schema.search.nl",), +``` + +```python + "nl": ("search", "nl"), +``` + +- [ ] **Step 6: Run the dependency test** + +Run: `uv run pytest ord_schema/dependencies_test.py -v` +Expected: PASS — five profiles, none skipped, since `anthropic` is installed in a development checkout. + +- [ ] **Step 7: Commit** + +```bash +git add ord_schema/search/nl.py ord_schema/search/nl_test.py pyproject.toml ord_schema/dependencies_test.py +git commit -m "Add the natural-language module, its extra, and its errors" +``` + +--- + +### Task 3: Translation, with the coercing parse + +**Files:** + +- Create: `ord_schema/search/nl_prompt.md` +- Modify: `ord_schema/search/nl.py`, `ord_schema/search/nl_test.py` + +**Interfaces:** + +- Consumes: `schema.describe()`, `query.Query`, `query.compile_query`, the errors from Task 2. +- Produces: `translate(question, *, client=None, model=DEFAULT_MODEL, repair=True) -> query.Query`; `SYSTEM_PROMPT: str`; `TOOL: dict`. + +- [ ] **Step 1: Write the failing tests** + +```python +class _StubClient: + """Returns canned tool calls, and records what it was asked.""" + + def __init__(self, *inputs): + self._inputs = list(inputs) + self.requests = [] + self.messages = self + + def create(self, **kwargs): + self.requests.append(kwargs) + payload = self._inputs.pop(0) + block = types.SimpleNamespace( + type="tool_use", id="toolu_stub", name="build_query", input=payload + ) + usage = types.SimpleNamespace( + input_tokens=1, output_tokens=1, + cache_creation_input_tokens=0, cache_read_input_tokens=0, + ) + return types.SimpleNamespace(content=[block], usage=usage, stop_reason="tool_use") + + +_WHERE = { + "op": "exists", + "path": "inputs.components", + "where": {"op": "eq", "path": "reaction_role", "value": {"literal": "SOLVENT"}}, +} + + +def test_a_tree_returned_as_a_json_string_is_still_understood(): + # Both models do this, most of the time: the predicate arrives JSON-encoded inside + # a string rather than as an object. + client = _StubClient({"where": json.dumps(_WHERE)}) + result = nl.translate("solvent reactions", client=client) + assert result.where.op == "exists" + + +def test_the_prefix_is_marked_cacheable(): + # Cache reads are most of what a query costs; an uncached prefix is a 10x bill. + client = _StubClient({"where": _WHERE}) + nl.translate("solvent reactions", client=client) + system = client.requests[0]["system"] + assert system[0]["cache_control"] == {"type": "ephemeral", "ttl": "1h"} + + +def test_the_schema_reaches_the_prompt(): + client = _StubClient({"where": _WHERE}) + nl.translate("solvent reactions", client=client) + assert "reaction_role" in client.requests[0]["system"][0]["text"] +``` + +- [ ] **Step 2: Run them** + +Run: `uv run pytest ord_schema/search/nl_test.py -v` +Expected: FAIL — `AttributeError: module 'ord_schema.search.nl' has no attribute 'translate'`. + +- [ ] **Step 3: Write the prompt** + +`ord_schema/search/nl_prompt.md`: + +```markdown +You turn a chemist's question into one ORD search query by calling `build_query`. + +Rules that keep a query answerable: + +- Paths are dotted names from the schema below. There is no array syntax: write + `inputs.components`, never `inputs[].components` or `identifiers[*].value`. +- Any path that crosses a repeated level must be bound by `exists` or `forall`, and the + paths inside that quantifier are relative to the bound element. +- Two conditions on the *same* element go inside one quantifier. Two conditions on + *different* elements are two quantifiers. +- Name compounds rather than spelling structures: `{"compound": "pyridine"}` resolves to + SMILES. Use `substructure` with a SMARTS only for a pattern the user describes. +- To rank by a value under a repeated level, order by a reduction: + `{"reduce": "max", "path": "outcomes.products.measurements.percentage.value"}`. +- Prefer the smallest query that answers the question, and set `limit` when the user + asks for a number of results. + +The corpus schema, as an indented type tree in DuckDB's types: +``` + +- [ ] **Step 4: Implement translation** + +```python +SYSTEM_PROMPT = ( + (resources.files("ord_schema.search") / "nl_prompt.md").read_text(encoding="utf-8") + + "\n\n" + + schema.describe() +) + +# Built once: the tool definition is part of the cached prefix, so a dict rebuilt per +# call with a different key order would silently cost a cache miss. +TOOL: dict[str, Any] = { + "name": "build_query", + "description": "Build an ORD search query from the user's question.", + "input_schema": query.Query.model_json_schema(), +} +_SYSTEM = [ + { + "type": "text", + "text": SYSTEM_PROMPT, + "cache_control": {"type": "ephemeral", "ttl": "1h"}, + } +] + + +def _coerce(value: Any) -> Any: + """Returns the value with JSON-encoded strings parsed back into objects. + + A model handed a recursive tool schema usually returns the nested predicate as a + JSON string rather than an object, which pydantic rejects. Parsing it back is the + normal path rather than a fallback. + """ + if isinstance(value, str): + try: + return _coerce(json.loads(value)) + except (json.JSONDecodeError, TypeError): + return value + if isinstance(value, dict): + return {key: _coerce(item) for key, item in value.items()} + if isinstance(value, list): + return [_coerce(item) for item in value] + return value + + +def _ask_model(client, model, messages): + """Returns the tool_use block from one forced call, mapping API failures.""" + try: + response = client.messages.create( + model=model, + max_tokens=MAX_TOKENS, + system=_SYSTEM, + messages=messages, + tools=[TOOL], + tool_choice={"type": "tool", "name": "build_query"}, + ) + except anthropic.RateLimitError as error: + raise ModelRateLimitedError(str(error)) from error + except (anthropic.APIConnectionError, anthropic.APIStatusError) as error: + raise ModelUnavailableError(str(error)) from error + for block in response.content: + if block.type == "tool_use": + return block + raise MalformedQueryError("the model returned no query") +``` + +- [ ] **Step 5: Add `translate` itself** + +```python +def translate( + question: str, + *, + client: anthropic.Anthropic | None = None, + model: str = DEFAULT_MODEL, + repair: bool = True, +) -> query.Query: + """Returns the query a question asks for. + + Args: + question: The question, in English. + client: Anthropic client; one is built from the environment if omitted. + model: Which model translates. Cheap models need the repair turn more often. + repair: Hand a failure back once with the compiler's error. Off for measuring + first-try accuracy. + + Returns: + A Query that compiles against the projection schema. + + Raises: + MalformedQueryError: If the query does not validate or compile, after the + repair turn if one was allowed. + ModelRateLimitedError: If the model is rate limited. + ModelUnavailableError: If the model cannot be reached. + """ + client = client if client is not None else get_client() + messages: list[dict[str, Any]] = [{"role": "user", "content": question}] + block = _ask_model(client, model, messages) + try: + return _validated(block.input) + except (ValueError, query.QueryError) as error: + if not repair: + raise MalformedQueryError(str(error)) from error + first = error + logger.info("repairing a query that did not compile: %s", first) + messages += [ + {"role": "assistant", "content": [{"type": "tool_use", "id": block.id, + "name": block.name, "input": block.input}]}, + {"role": "user", "content": [{"type": "tool_result", "tool_use_id": block.id, + "is_error": True, + "content": f"That query was rejected: {first}. " + "Call build_query again with it fixed."}]}, + ] + retry = _ask_model(client, model, messages) + try: + return _validated(retry.input) + except (ValueError, query.QueryError) as error: + raise MalformedQueryError(str(error)) from error + + +def _validated(raw: Any) -> query.Query: + """Returns the Query a tool call carries, proven to compile.""" + parsed = query.Query.model_validate(_coerce(raw)) + query.compile_query(parsed) + return parsed +``` + +- [ ] **Step 6: Run the tests** + +Run: `uv run pytest ord_schema/search/nl_test.py -v` +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add ord_schema/search/nl.py ord_schema/search/nl_prompt.md ord_schema/search/nl_test.py +git commit -m "Translate a question into a query, coercing what the model returns" +``` + +--- + +### Task 4: The repair turn, pinned + +**Files:** + +- Modify: `ord_schema/search/nl_test.py` + +**Interfaces:** + +- Consumes: `translate`, `_StubClient` from Task 3. + +- [ ] **Step 1: Write the failing tests** + +```python +_BAD_PATH = {"op": "eq", "path": "identifiers[*].value", "value": {"literal": "x"}} + + +def test_a_bad_path_is_handed_back_once_and_recovered(): + client = _StubClient({"where": _BAD_PATH}, {"where": _WHERE}) + result = nl.translate("aspirin reactions", client=client) + assert result.where.op == "exists" + assert len(client.requests) == 2 + + +def test_the_repair_carries_the_compiler_s_suggestion(): + client = _StubClient({"where": _BAD_PATH}, {"where": _WHERE}) + nl.translate("aspirin reactions", client=client) + sent = client.requests[1]["messages"][-1]["content"][0]["content"] + assert "did you mean" in sent + + +def test_a_second_failure_raises_rather_than_looping(): + client = _StubClient({"where": _BAD_PATH}, {"where": _BAD_PATH}) + with pytest.raises(nl.MalformedQueryError, match="identifiers"): + nl.translate("aspirin reactions", client=client) + assert len(client.requests) == 2 + + +def test_repair_can_be_turned_off_for_measurement(): + client = _StubClient({"where": _BAD_PATH}) + with pytest.raises(nl.MalformedQueryError): + nl.translate("aspirin reactions", client=client, repair=False) + assert len(client.requests) == 1 +``` + +- [ ] **Step 2: Run them** + +Run: `uv run pytest ord_schema/search/nl_test.py -k repair -v` +Expected: PASS if Task 3 was implemented as written; a failure here means the repair path is wrong, not that the test is. + +- [ ] **Step 3: Commit** + +```bash +git add ord_schema/search/nl_test.py +git commit -m "Pin the repair turn to exactly one attempt" +``` + +--- + +### Task 5: The summary and the prose answer + +**Files:** + +- Modify: `ord_schema/search/nl.py`, `ord_schema/search/nl_test.py` + +**Interfaces:** + +- Produces: `summarize(table, *, rows=5) -> str`; `answer(question, table, *, client=None, model=DEFAULT_MODEL) -> str`. + +- [ ] **Step 1: Write the failing tests** + +```python +def test_the_summary_is_bounded_by_the_row_cap_not_the_table(): + small = pa.table({"reaction_id": ["a", "b"]}) + large = pa.table({"reaction_id": [str(i) for i in range(100_000)]}) + assert len(nl.summarize(large)) < 2 * len(nl.summarize(small)) + + +def test_the_summary_states_the_row_count(): + assert "100000 rows" in nl.summarize(pa.table({"reaction_id": [str(i) for i in range(100_000)]})) +``` + +- [ ] **Step 2: Run them** + +Run: `uv run pytest ord_schema/search/nl_test.py -k summary -v` +Expected: FAIL — `summarize` does not exist. + +- [ ] **Step 3: Implement both** + +```python +def summarize(table: pa.Table, *, rows: int = 5) -> str: + """Returns a description of a result small enough to put in a prompt. + + Args: + table: What the search returned. + rows: How many sample rows to show. + + Returns: + The row count, the column names, and up to ``rows`` rows. A result of a hundred + thousand reactions costs the same prompt as one of three. + """ + sample = table.slice(0, rows).to_pylist() + lines = [f"{table.num_rows} rows, columns: {', '.join(table.column_names)}"] + lines += [json.dumps(row, default=str) for row in sample] + if table.num_rows > rows: + lines.append(f"... {table.num_rows - rows} more rows not shown") + return "\n".join(lines) + + +def answer( + question: str, + table: pa.Table, + *, + client: anthropic.Anthropic | None = None, + model: str = DEFAULT_MODEL, +) -> str: + """Returns a sentence or two saying what the result shows. + + Args: + question: The question the result answers. + table: What the search returned. + client: Anthropic client; one is built from the environment if omitted. + model: Which model writes the prose. + + Returns: + Plain text, with no markdown and no invented chemistry: the model sees a summary + rather than the rows, so it can only describe what the summary states. + + Raises: + ModelRateLimitedError: If the model is rate limited. + ModelUnavailableError: If the model cannot be reached. + """ + client = client if client is not None else get_client() + try: + response = client.messages.create( + model=model, + max_tokens=512, + system=( + "You describe the result of a database query in one or two plain " + "sentences. State only what the summary shows. Do not invent " + "chemistry, and do not use markdown." + ), + messages=[ + { + "role": "user", + "content": f"Question: {question}\n\nResult:\n{summarize(table)}", + } + ], + ) + except anthropic.RateLimitError as error: + raise ModelRateLimitedError(str(error)) from error + except (anthropic.APIConnectionError, anthropic.APIStatusError) as error: + raise ModelUnavailableError(str(error)) from error + return "".join(block.text for block in response.content if block.type == "text") +``` + +- [ ] **Step 4: Run the tests** + +Run: `uv run pytest ord_schema/search/nl_test.py -k summary -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add ord_schema/search/nl.py ord_schema/search/nl_test.py +git commit -m "Summarize a result and let the model describe it" +``` + +--- + +### Task 6: The round trip + +**Files:** + +- Modify: `ord_schema/search/nl.py`, `ord_schema/search/nl_test.py` + +**Interfaces:** + +- Produces: `Answer(question, query, table, text)`; `ask(question, corpus, *, client=None, model=DEFAULT_MODEL, timeout_seconds=60.0) -> Answer`. + +- [ ] **Step 1: Write the failing test** + +```python +def test_ask_returns_the_query_it_ran(corpus): + # The caller needs the query to show what was actually asked, and to offer a rerun. + client = _StubClient({"where": _WHERE}, _text("Two reactions, both with pyridine.")) + result = nl.ask("solvent reactions", corpus, client=client) + assert result.query.where.op == "exists" + assert result.table.num_rows == result.table.num_rows + assert "pyridine" in result.text +``` + +Extend `_StubClient` so a canned entry that is a string becomes a text response: + +```python +def _text(value: str): + return value + + +# in _StubClient.create, before building the tool_use block: + if isinstance(payload, str): + block = types.SimpleNamespace(type="text", text=payload) + return types.SimpleNamespace(content=[block], usage=usage, stop_reason="end_turn") +``` + +- [ ] **Step 2: Run it** + +Run: `uv run pytest ord_schema/search/nl_test.py -k ask -v` +Expected: FAIL — `ask` does not exist. + +- [ ] **Step 3: Implement it** + +```python +@dataclasses.dataclass(frozen=True) +class Answer: + """What a question produced, including the query it became. + + Attributes: + question: The question as asked. + query: The query that ran, for display and for running again. + table: What the search returned. + text: A sentence or two describing the result. + """ + + question: str + query: query.Query + table: pa.Table + text: str + + +def ask( + question: str, + corpus: execute.Corpus, + *, + client: anthropic.Anthropic | None = None, + model: str = DEFAULT_MODEL, + timeout_seconds: float = 60.0, +) -> Answer: + """Answers a question against a corpus. + + Args: + question: The question, in English. + corpus: The corpus to search. + client: Anthropic client; one is built from the environment if omitted. + model: Which model translates and describes. + timeout_seconds: Passed to the search, so a slow query fails rather than hangs. + + Returns: + The query, the reactions, and a description of them. + + Raises: + MalformedQueryError: If translation does not produce a query that compiles. + ModelRateLimitedError: If the model is rate limited. + ModelUnavailableError: If the model cannot be reached. + """ + client = client if client is not None else get_client() + translated = translate(question, client=client, model=model) + table = corpus.search(translated, timeout_seconds=timeout_seconds) + return Answer( + question=question, + query=translated, + table=table, + text=answer(question, table, client=client, model=model), + ) +``` + +- [ ] **Step 4: Run the whole file** + +Run: `uv run pytest ord_schema/search/nl_test.py -v` +Expected: PASS, and no test makes a network call. + +- [ ] **Step 5: Document it** + +Add to `ord_schema/search/README.md`, under `## Usage`: + +````markdown +### Ask in English + +```python +from ord_schema.search import execute, nl + +corpus = execute.Corpus("projections/*/*.parquet", "structures/*/*.parquet") +answer = nl.ask("which reactions use pyridine as a solvent?", corpus) +print(answer.query, answer.table.num_rows, answer.text) +``` + +Needs the `nl` extra (`pip install "ord-schema[nl]"`) and `ANTHROPIC_API_KEY`. The model +cannot be constrained to emit a valid query — the grammar is recursive and both +constrained-decoding paths refuse it — so a failure is handed back once with the +compiler's own error, and a second failure raises `MalformedQueryError`. +```` + +- [ ] **Step 6: Commit** + +```bash +git add ord_schema/search/nl.py ord_schema/search/nl_test.py ord_schema/search/README.md +git commit -m "Answer a question end to end" +``` + +--- + +### Task 7: The eval harness + +**Files:** + +- Create: `ord_schema/search/nl_eval.py`, `ord_schema/search/nl_cases.yaml`, `ord_schema/search/nl_eval_test.py` + +**Interfaces:** + +- Consumes: `translate`, `execute.Corpus`. +- Produces: `EvalCase`, `load_cases(path) -> list[EvalCase]`, `run_case(case, corpus, *, client, model, repair) -> CaseResult`, `report(results) -> str`. + +- [ ] **Step 1: Write the failing test** + +```python +def test_a_case_that_returns_a_forbidden_reaction_fails(corpus): + # Scoring on reactions rather than on query shape is what lets the harness say a + # translation is wrong rather than merely differently spelled. + case = nl_eval.EvalCase( + question="solvent reactions", + must_return=["ord-aa000000000000000000000000000000"], + must_not_return=["ord-bb000000000000000000000000000000"], + ) + result = nl_eval.score(case, returned=["ord-bb000000000000000000000000000000"]) + assert not result.passed + assert "must_not_return" in result.detail +``` + +- [ ] **Step 2: Run it** + +Run: `uv run pytest ord_schema/search/nl_eval_test.py -v` +Expected: FAIL — module does not exist. + +- [ ] **Step 3: Write the harness** + +```python +class EvalCase(BaseModel): + """One question and what any correct answer to it must satisfy. + + Attributes: + question: The question, in English. + must_return: Reaction IDs any correct query returns. + must_not_return: Reaction IDs a near-miss would wrongly include. + compiles: Whether the question should translate at all; False marks a question + the grammar cannot express, which the layer must refuse rather than fudge. + """ + + question: str + must_return: list[str] = Field(default_factory=list) + must_not_return: list[str] = Field(default_factory=list) + compiles: bool = True + + +@dataclasses.dataclass(frozen=True) +class CaseResult: + """How one case came out.""" + + case: EvalCase + passed: bool + detail: str + + +def score(case: EvalCase, returned: Sequence[str]) -> CaseResult: + """Returns whether the reactions a query returned satisfy the case.""" + found = set(returned) + missing = [value for value in case.must_return if value not in found] + forbidden = [value for value in case.must_not_return if value in found] + if missing: + return CaseResult(case, False, f"must_return absent: {missing}") + if forbidden: + return CaseResult(case, False, f"must_not_return present: {forbidden}") + return CaseResult(case, True, f"{len(found)} reactions") + + +def load_cases(path: str) -> list[EvalCase]: + """Returns the cases a YAML file holds.""" + with pathlib.Path(path).open(encoding="utf-8") as handle: + return [EvalCase.model_validate(entry) for entry in yaml.safe_load(handle)] +``` + +- [ ] **Step 4: Add the runner and the report** + +```python +def run_case( + case: EvalCase, + corpus: execute.Corpus, + *, + client: anthropic.Anthropic, + model: str, + repair: bool, +) -> CaseResult: + """Translates and runs one case, returning how it scored.""" + try: + translated = nl.translate(case.question, client=client, model=model, repair=repair) + except nl.MalformedQueryError as error: + passed = not case.compiles + return CaseResult(case, passed, f"did not compile: {error}") + if not case.compiles: + return CaseResult(case, False, "compiled, but the case expects it cannot") + table = corpus.search(translated) + return score(case, table.column("reaction_id").to_pylist()) + + +def report(results: Sequence[CaseResult]) -> str: + """Returns a human-readable summary, failures first.""" + passed = sum(result.passed for result in results) + lines = [f"{passed}/{len(results)} passed"] + lines += [ + f" FAIL {result.case.question}: {result.detail}" + for result in results + if not result.passed + ] + return "\n".join(lines) +``` + +- [ ] **Step 5: Write the starting cases** + +`ord_schema/search/nl_cases.yaml`, with reaction IDs filled in by running each query by hand against the local corpus first: + +```yaml +# Each case states what any correct translation must return, never a Query literal: +# several spellings are right, and pinning one would fail a better query than the one +# that was written when the case was added. +- question: reactions using pyridine as the solvent + must_return: [] + must_not_return: [] +- question: reactions run above 350 K + must_return: [] + must_not_return: [] +- question: the ten highest-yielding reactions + must_return: [] + must_not_return: [] +``` + +- [ ] **Step 6: Run the tests** + +Run: `uv run pytest ord_schema/search/nl_eval_test.py -v` +Expected: PASS. + +- [ ] **Step 7: Take the first real measurement** + +Run, with a key in the environment and the local corpus: + +```bash +uv run python -m ord_schema.search.nl_eval --model claude-haiku-4-5 --repair +uv run python -m ord_schema.search.nl_eval --model claude-opus-5 --repair +``` + +Record both in the logbook entry rather than in a commit message: the numbers in +finding 6 came from ten ad-hoc questions and want replacing with these. + +- [ ] **Step 8: Commit** + +```bash +git add ord_schema/search/nl_eval.py ord_schema/search/nl_cases.yaml ord_schema/search/nl_eval_test.py +git commit -m "Score translations on the reactions they return" +``` + +--- + +## Self-review + +**Spec coverage.** `ask`/`translate`/`answer` — Tasks 3, 5, 6. Cached prefix — Task 3, pinned by a test. Coercion — Task 3. One repair turn — Tasks 3 and 4. Summary not table — Task 5. Module layout and the `nl` extra — Task 2. Error taxonomy — Task 2. Eval harness scoring on reactions — Task 7. The reduction gap — Task 1. Out of scope in the spec and absent here: multi-turn, a second backend, constrained decoding, text-search tuning. + +**Types.** `translate` returns `query.Query` in Tasks 3, 6, and 7. `Answer.query` is that same type. `summarize` and `answer` both take `pa.Table`, which is what `Corpus.search` returns. `Reduction` is referenced by `Order.key` and `Measure.path` only. + +**Known gaps, deliberately left to the executor.** The eval cases in Task 7 ship with empty `must_return` lists, because the IDs have to come from running each query against the local corpus — filling them in is step 5's work, and inventing IDs here would be worse than leaving them empty. Task 1's step 6 needs the local projections; it is a check, not a test, and does not run in CI. From 02184ed5923b5f0b3ff1d69a0eed10ac0a2cd99d Mon Sep 17 00:00:00 2001 From: Steven Kearnes Date: Mon, 17 Aug 2026 22:29:45 -0400 Subject: [PATCH 3/3] Record what implementing the layer changed Three things the design got wrong, each found by building it: a resolved compound compared in PubChem's spelling rather than the corpus's, which answered a question about 24,930 reactions with zero; a forced tool call that left the model no way to decline, so both models built a plausible query for a question the grammar cannot express; and an eval case whose counterexamples were sampled rather than derived, which failed both models on a translation they had got right. Co-Authored-By: Claude Opus 5 (1M context) --- .../README.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md b/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md index 7a218a8..4c7901d 100644 --- a/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md +++ b/entries/2026-08-17-what-constrains-a-natural-language-layer/README.md @@ -191,6 +191,36 @@ Three things to do in order: 3. **Then choose the model on evidence**, including whether path guidance in the cached prefix closes Haiku's gap. At 5.6× it is worth the measurement. +## What implementing it changed + +Three things the design got wrong, found by building it (ord-schema +[#979](https://github.com/open-reaction-database/ord-schema/pull/979), +[#980](https://github.com/open-reaction-database/ord-schema/pull/980), +[#981](https://github.com/open-reaction-database/ord-schema/pull/981)): + +**A resolved compound was compared in the wrong spelling.** The default resolver asks +PubChem, which returns pyridine as `C1=CC=NC=C1`; the projection stores what RDKit +canonicalizes, `c1ccncc1`. Compared as strings they never match, so the first real +question -- pyridine as a solvent -- answered **zero** over a corpus holding 24,930. +Wrong, and wrong in the way that looks like an answer. This predates the NL layer: any +caller passing `{"compound": ...}` to an `eq` on a smiles path hit it. + +**Forcing the tool call left no way to decline.** With `tool_choice` naming +`build_query`, a model asked for something the grammar cannot express -- comparing two +columns -- builds a plausible query that means something else. Both models did. A +`cannot_answer` tool beside it, with `tool_choice: any`, turns that into a refusal +carrying the model's own reason. + +**An eval case built by sampling failed correct translations.** The counterexamples for +"a desired product with a yield above 50%" were derived by differencing a 200-row sample +of the right query against a 400-row sample of the wrong one -- set arithmetic on +samples. Both models failed a case they had answered correctly, and a round of prompt +tuning went into a problem that did not exist. Asking for the counterexamples directly +-- `near_miss AND NOT reference` -- is exact, and with honest cases both models pass 5/5. + +The second and third are the same lesson from opposite directions: a layer that cannot +say "no" invents an answer, and a test that cannot say "wrong" invents a failure. + ## References - [`assets/nl-search-design.md`](assets/nl-search-design.md) — the design this entry argues for