Record what constrains a natural-language layer, and the design it forces - #45
Open
skearnes wants to merge 3 commits into
Open
Record what constrains a natural-language layer, and the design it forces#45skearnes wants to merge 3 commits into
skearnes wants to merge 3 commits into
Conversation
…rces Constrained decoding cannot carry the search grammar: circular refs are refused, the compiled grammar has a size budget, oneOf is unsupported, and union-typed parameters are capped. Stratifying into acyclic levels, requiring every property, and flattening the predicate union each moved a wall, and the best that fits is a depth-2 tree -- which cannot express the correlation the pivots exist to serve. So translation is unconstrained generation checked afterwards, which the measurements say is fine: Opus 5 compiles 9/10 first try, Haiku 4.5 gets 7/10 with one repair turn at 5.6x less. The design beside the entry builds on that, and starts by closing a gap the probing exposed -- an aggregate cannot reach a value under a repeated level, so "the ten highest-yielding reactions" is unwritable and both models tried anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven tasks: the reduction that closes the grammar gap first, then the module and its extra, translation with the coercing parse, the repair turn pinned to one attempt, the summary and prose answer, the round trip, and an eval harness that scores on the reactions a query returns rather than on how the query is spelled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 18, 2026
Three things the design got wrong, each found by building it: a resolved compound compared in PubChem's spelling rather than the corpus's, which answered a question about 24,930 reactions with zero; a forced tool call that left the model no way to decline, so both models built a plausible query for a question the grammar cannot express; and an eval case whose counterexamples were sampled rather than derived, which failed both models on a translation they had got right. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An entry recording why the NL layer cannot use constrained decoding, and a design doc beside it for the layer that follows. The measurements are live API calls, not inference from docs — the probes are in the entry's assets.
Changes
entries/2026-08-17-what-constrains-a-natural-language-layer/README.md— seven findings: the four walls constrained decoding hits, what a flattened depth-2 grammar with all 431 scalar paths enumerated does fit, the JSON-string shape both models return, model accuracy and cost, and a gap in the grammar itself.assets/nl-search-design.md— the design for review: scope, module layout, the translate/repair/answer flow, error taxonomy, testing, and three open questions.assets/*.py— twelve probes, including two reusable schema transforms (stratify.py,require_all.py).assets/nl-search-plan.md— seven tasks, TDD throughout, each ending in a commit.Testing
Every table in the entry is a live call against the Messages API using the interface deployment's key.
probe_repair.py <model>reproduces the accuracy and cost figures;probe_recursion.pyneeds no key at all, since it uses a mock transport to capture what the SDK puts on the wire.Notes
The plan is now beside the design. Task 1 is the grammar change, not NL code; tasks 2-7 build the layer and its eval harness.
The design is draft, for review — please read
assets/nl-search-design.mdbefore I turn it into an implementation plan. The part most worth your eye is "The gap this depends on": aggregates and ordering keys cannot reach a value under a repeated level, sothe ten highest-yielding reactionsis unwritable today. I propose closing that in the grammar first, since an NL layer built over it would answer the question wrongly rather than refuse.Two numbers that looked inconsistent are reconciled in the entry: the probe walks to 431 scalar paths, while the schema rendering counts 442 leaves, because the rendering carries the key side of a map too.
🤖 Generated with Claude Code