Score translations on the reactions they return

[skearnes]

Summary

The eval harness (task 7), plus a gap it exposed and closed. A case states reactions any correct query returns and reactions a plausible wrong one returns — never the query it expects, since several spellings of a question are right.

$ python -m ord_schema.search.nl_eval --projections ... --structures ... --model claude-haiku-4-5
5/5 passed

Both claude-haiku-4-5 and claude-opus-5 pass 5/5 against the full corpus.

Two things worth your attention

My first version of the cases was wrong, and it cost a round of prompt tuning. I built must_not_return by differencing a 200-row sample of the reference against a 400-row sample of the near-miss — set arithmetic on samples. It failed correct translations on both models, and I "fixed" the prompt for a problem that did not exist before checking the case itself. Counterexamples are now found by asking for them — near_miss AND NOT reference — which is exact, and the builder raises if a near-miss excludes nothing. The prompt example I added stands on its own merits but changed no measurement, and the commit says so.

Forcing build_query left the model no way to decline. Asked for something the grammar cannot express — comparing two columns — both models built a plausible query that means something else. That is the failure mode that looks most like an answer. cannot_answer is now offered beside build_query with tool_choice: any, declining raises UnanswerableError carrying the model's reason, and a refusal is never repaired since nothing was wrong with its reasoning. With that, the inexpressible case passes on both models by being refused.

Changes

• nl_eval.py — EvalCase, score, run_case, report, and a command. Every case carries a why, printed with any failure.
• nl_cases_build.py — how the IDs were derived, rerunnable when the corpus changes. IDs are only meaningful against the corpus they came from, which the module says.
• nl_cases.json — five cases: the scoping trap, a scalar comparison, correlation, a forall, and one the grammar cannot express.
• nl.py — REFUSAL_TOOL, UnanswerableError.
• The eval command takes --pivots-dir and --pivot-budget-bytes, defaulting the budget to 0: a run is a handful of queries, and building a pivot over 2.4M reactions costs minutes to save milliseconds. Without that the first run hung.

Testing

uv run pytest -n auto — 1246 passed, none reaching the network. The harness tests cover scoring, the report, and that every shipped case has both a reason and something to assert; the refusal tests cover the new tool, the error, and that a refusal costs one request rather than two.

Notes

Deviation from the plan: cases are JSON, not YAML. PyYAML is installed here but undeclared, and importing it would have been exactly what dependencies_test.py exists to catch. Each case carries a why field instead of a comment, which is better anyway — it prints with the failure.

dependencies_test.py also caught nl_eval importing anthropic while belonging to the search profile; its roots now name both modules.

Stacked on #980 → #979; retarget as those land.

🤖 Generated with Claude Code

Greptile Summary

This PR adds explicit refusal support to natural-language query translation and an evaluation harness that scores translations by returned reaction IDs.

• Offers cannot_answer alongside build_query and exposes refusals through UnanswerableError.
• Adds five corpus-derived evaluation cases, their reproducible builder, reporting, CLI options, and tests.
• Expands the prompt and documentation around correlated quantifiers and inexpressible questions.
• Packages the JSON cases and updates dependency-profile coverage.

Confidence Score: 4/5

The evaluation harness should be corrected before merging because it can report malformed model output as a successful refusal.

The production refusal path is coherently separated from malformed queries, but run_case collapses those outcomes and can therefore conceal a regression in the newly added evaluation.

Files Needing Attention: ord_schema/search/nl_eval.py

Important Files Changed

Filename	Overview
ord_schema/search/nl.py	Adds a refusal tool and a distinct exception while preserving the existing validation and repair flow.
ord_schema/search/nl_eval.py	Adds the evaluation runner and reporting, but incorrectly accepts malformed translation failures as successful refusals.
ord_schema/search/nl_cases_build.py	Builds positive examples and exact near-miss counterexamples from a supplied corpus.
ord_schema/search/nl_cases.json	Defines four executable semantic cases and one explicitly inexpressible case.
ord_schema/search/nl_eval_test.py	Covers scoring and fixture validation but does not exercise run_case exception semantics.
ord_schema/search/nl_prompt.md	Documents when to refuse and clarifies correlated quantifier construction.
pyproject.toml	Includes JSON evaluation fixtures in the packaged search module.

Sequence Diagram

sequenceDiagram
    participant Eval as nl_eval.run_case
    participant NL as nl.translate
    participant Model as Anthropic model
    participant Corpus as Search corpus
    Eval->>NL: Translate case question
    NL->>Model: Offer build_query or cannot_answer
    alt cannot_answer
        Model-->>NL: Explicit refusal
        NL-->>Eval: UnanswerableError
        Eval-->>Eval: "Pass only if compiles=false"
    else valid build_query
        Model-->>NL: Query
        NL-->>Eval: Validated query
        Eval->>Corpus: Execute query
        Corpus-->>Eval: Reaction IDs
        Eval-->>Eval: Score required/forbidden IDs
    else malformed build_query
        Model-->>NL: Invalid query
        NL-->>Eval: MalformedQueryError
        Eval-->>Eval: "Currently passes when compiles=false"
    end

Loading

_{Reviews (1): Last reviewed commit: "Score translations on the reactions they..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[greptile-apps]

Malformed output passes refusal case

When an inexpressible case produces an invalid build_query call instead of cannot_answer, run_case marks the resulting MalformedQueryError as passing, causing the evaluation to hide a failure to refuse.

Suggested change

	except nl.MalformedQueryError as error:
	return CaseResult(
	case, passed=not case.compiles, detail=f"did not compile: {error}"
	)
	except nl.MalformedQueryError as error:
	return CaseResult(
	case, passed=False, detail=f"did not compile: {error}"
	)