SPARQL: plan a query once, and answer aggregates from SQL where it is sound - #28
Merged
Conversation
extract_top_level_limit recurses through Slice and Project only, and the
Phase-4 guard checked stars/joins/OPTIONAL but not the solution modifiers.
So a LIMIT was pushed into the object fetch for queries where an operator
has to see every solution first, and the result was a plausible wrong
answer with no error:
SELECT ?v (COUNT(*) AS ?n) WHERE { ?s a :Signal ; :status ?v }
GROUP BY ?v LIMIT 10
counted ten objects rather than counting all of them and returning ten
groups. ORDER BY ... LIMIT sorted an arbitrary ten rows instead of the
top ten, and SELECT DISTINCT ... LIMIT deduplicated an arbitrary ten,
returning fewer distinct values than exist — the shape a filter dropdown
uses.
has_holistic_modifier walks the pattern for Group (which covers both
GROUP BY and bare aggregates, since spargebra models
`SELECT (COUNT(*) AS ?n)` as a Group with no grouping variables), OrderBy,
Distinct and Reduced. Its match is left exhaustive so a future
GraphPattern variant is a compile error rather than a silent "safe to
push down".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
New sparql_pushdown module. Where sparql_scoper decides what to *load* so oxigraph can answer a query, this decides whether a query's grouping and aggregation can be answered by SQL without loading anything — the only shape that works for an aggregate, which by definition touches every object of its class. The verdict is three-way on purpose: "not an aggregate" and "an aggregate outside the supported subset" need different handling and must not collapse into one falsy value. The first keeps the existing route silently; the second is reportable to whoever wrote the query, so every refusal carries a BlockedCode from a closed set plus where/why and a rewrite hint. The intent is that the boundary of the subset can be learned from the errors rather than from a document. SolutionSpec addresses result columns by position, never by SPARQL variable name: those come from the request, and unquoted Postgres identifiers case-fold and truncate at 63 bytes, so two distinct variables could otherwise collide into one column. Measure carries its argument inside the variant, making COUNT(*)-with-an-argument unrepresentable. TermRef is a pointer into the schema rather than a copy of the rendering decision, and `numeric` is decided here because it is schema knowledge the renderer needs twice over (cast before aggregation; COLLATE "C" for text comparison). Two things spargebra's shapes forced, both worth knowing: - `(COUNT(*) AS ?n)` is modelled as a Group binding an internal variable plus an Extend aliasing it, so "has an Extend" cannot mean "computes a value". Aliases resolve to the query's own name; real computations are refused. - BIND sits *under* the Group (it happens before grouping), so peeling the outer modifiers never sees it. Detected there, or the refusal names the wrong problem. Also exposes Star.slot_variables — the variable→slot mapping the star decomposition already computes to find join edges. It answers "which column does ?name come from" without re-walking the query, which both the pushdown analyser and a column projection need. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kervel
force-pushed
the
feat/sparql-aggregate-pushdown
branch
from
August 21, 2026 20:06
5202f64 to
01e95b9
Compare
The SQL pushdown route reads text out of JSONB; to answer in SPARQL it must turn that text into the same term the oxigraph route would produce, or one question answers differently depending on which route served it. The decision depends only on the slot, so it is resolved once per column at plan time and applied per row by the renderer. New sparql_terms module. `term_descriptor(schema_view, class_uri, slot_path)` returns the term kind (iri / literal / enum_iri), the datatype IRI, a language tag, the permissible-value → IRI map for enums, and whether the values are numbers. Pull-based on an explicit path because a schema may be cyclic — a class with a slot of its own range has no finite enumeration of paths to precompute — while a path taken from a query is always finite. This replaces BindingSpec.numeric, which compared the range's *name* against a hardcoded list of LinkML type names. Deciding numeric-ness from the resolved datatype instead keeps the two routes in step by construction: the turtle writer picks typed-vs-plain from the same field, so a value treated as a number here is exactly one oxigraph sees as a numeric literal. Known limitation, asserted by a test rather than papered over: a type declared only as `typeof: integer`, with no `uri` of its own, resolves to no datatype at all — so it is a plain literal to the writer and non-numeric here, and SUM/AVG over it are refused. Conservative rather than wrong (summing plain literals is a type error in SPARQL too), and the fix belongs upstream in determine_rdf_type_info, where it also changes serialisation. It does not arise in practice: asset360's schemas use the bundled LinkML types, which all carry a uri. The scoper's test schema now declares its types locally. Without them `range: integer` resolved to no datatype, which would have made every slot look non-numeric — the fixture was hiding the very thing these descriptors decide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FILTER(?len > 10) and its relatives now reach the WHERE clause instead of
being left for oxigraph, which matters most for the pushdown route: a
report filtered to a range should scan the range, not the class.
FilterCondition gains Cmp { op, value } rather than a new predicate
language — this is the representation the SPARQL path already uses, and
the repo's other predicate IRs (Rust `Predicate`, advanced_filters) come
with ORM translators that do not apply to raw SQL.
Deliberately not included: `!=`. SPARQL's inequality is false for an
unbound variable, where SQL's `<>` against NULL is unknown and would drop
rows the query keeps.
A comparison written the other way round (`10 < ?len`) flips the operator,
or the filter would exclude exactly the rows it should keep. Tested.
An ordering comparison on the identifier slot cannot be hoisted into the
`asset360_uri IN (...)` form the way equality is — there is no finite value
list — so it stays a filter and the renderer targets the same indexed
column with the operator. The exhaustive matches over FilterCondition
found both hoisting sites, which is why they are written that way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`?s :location ?l . ?l :longitude ?v` binds ?v two slots down from ?s, and no star can describe it: ?l has no rdf:type and is part of ?s's JSON rather than an object of its own. The scoper already walks those triples to find join edges, so it now also records where such leaves live — QueryPlan.path_bindings maps a variable to (star, slot path) — and the pushdown analyser resolves a group key or measure argument through them. The renderer already read a multi-slot path. Which slots are walked matters. Only *inlined* ranges: a reference stores the target's URI, so there is no nested data to read through, and treating it as a JSON path would look for something that is not there. Cyclic schemas are handled with a depth bound rather than a visited set, because revisiting a class is legitimate — `?a :child ?b . ?b :child ?c` is a real query — while unbounded depth is not. Four is well past anything the data holds. A bug this surfaced, and the distinction it turns on: grouping by ?loc itself was accepted and described as a *literal*, so SQL would have returned the whole nested JSON as text where oxigraph returns a blank node. The two class-ranged cases are not the same: - a *reference* stores the target's URI, which is exactly the IRI oxigraph binds, so grouping by it answers "how many per parent" — a report people actually want, and now supported; - an *inlined* structure has no reproducible identity (a blank node label is not stable even between two oxigraph runs), so it is traversable but never a value, and grouping by it is refused with a hint. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"How many components per tunnel complex" spans two classes joined on a reference, and the group key belongs to one of them while the counted rows belong to the other. The analyser now accepts that: a binding names its own star, so the renderer reads each column from the right table alias, and the existing join edges become SQL JOINs. Two shapes stay refused, for reasons worth keeping: - classes with no reference between them. The SQL would be a cross product where oxigraph evaluates two independent patterns — different questions, so answering either would be wrong. - a class introduced inside OPTIONAL. A left-joined star contributes unbound rows, which changes what the aggregates count; supported for values *inside* a star, not yet for an optional star of its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RDF gives one triple per element of a collection, so a query over a multivalued slot has one solution per element. A consumer that reads the array as a whole reports one row per record instead — a wrong count with no error, and the kind of wrongness that looks plausible. BindingSpec gains `containers`, parallel to `slot_path`: how each step is stored, as single / list / mapping. Per step because any hop along a path may be a collection, and each one multiplies the rows. The names mirror the schema's own SlotContainerMode rather than inventing a fourth vocabulary for the same three-way distinction. Mapping is included because the turtle writer iterates a mapping's *values* — its keys are not part of the graph — so a consumer unnests it with jsonb_each and ignores the keys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…repr
Four review findings.
**Parser mismatch.** analyse_pushdown built a bare SparqlParser while
sparql_scope used one preloading asset360/rdf/rdfs/xsd. A query leaving
those implicit scoped fine and then failed to parse in the analyser — and
since the caller does not catch ValueError, that surfaced as a 500 on a
query that had worked. Every test in the new module prepends PREFIX, which
is exactly how this stayed hidden; there is now a test that does not.
Both entry points share sparql_parser(), and the parsed query is passed to
the new scope_parsed(), so the pushdown path parses once instead of three
times (bare parse, throwaway parse_update, then sparql_scope re-parsing).
Update rejection moved into parse_query, so an Update is still reported as
an Update rather than as a syntax error.
**Numeric-ness duplicated upstream.** XSD_NUMERIC reimplemented what
RangeInfo::is_integer()/is_floating_point() already do, and disagreed with
them: the list caught xsd:int/long/short/unsigned*, upstream catches
schemas whose types are unresolved via a builtin-name fallback. That
disagreement had already cost something — the shared scoper fixture had
grown a local `types:` block to work around the missing fallback, and this
commit reverts it. Now using the upstream helpers.
Left as a gap rather than worked around, per AGENTS.md: neither helper
covers xsd:int / long / short / unsigned*, so a slot declared with one is
treated as non-numeric — conservative (SUM/AVG refused, never miscast).
The fix is a RangeInfo::is_numeric() request upstream.
**The LIMIT fix left the bug's shape.** The original bug was limit
extraction and the eligibility guard disagreeing about which modifiers
matter, and adding a second independent walk kept that possible while
walking the pattern twice. pushable_limit() now owns the whole question:
finding a limit and refusing to push it are one decision, so there is no
way to get a limit out without passing the check.
**FilterCondition repr and docs.** The new comparison operators were
absent from the class doc, the getter doc and __repr__, so
Cmp { Gt, "10" } printed as FilterCondition(in=["10"]) — the repr branched
on "eq" and labelled everything else "in". It now prints the operator, and
the docs list all six with a note that an unknown one must be refused
rather than dropped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The systemic bug behind the last round of findings, and the one that made
them all instances rather than coincidences.
sparql_scoper exists to decide what to *load*. Every extraction step in it
is deliberately lossy in the safe direction: a constraint it cannot express
is dropped, the fetch widens, and oxigraph re-applies the real query to
whatever came back. Nothing is lost because something else checks.
analyse_pushdown reused those same outputs as an exact plan — the SQL *is*
the answer, nothing re-checks it — and never asked whether what the scoper
dropped was nothing. Same lossiness, but now it is a plausible wrong number
with no error:
- FILTER(?nm != "x"), REGEX, ||, !, BOUND, var-to-var comparisons all fell
off a silent `_ => {}`, so `COUNT(*)` counted every row of the class.
- Half of a conjunction landing is still a weaker filter.
- A FILTER inside OPTIONAL was pushed into the fetch, dropping rows the
LEFT JOIN exists to preserve. Inline constants are depth-gated for
exactly this reason; filter collection recursed into LeftJoin with no
gate. spargebra also lifts `OPTIONAL { ... FILTER(...) }` into the
LeftJoin's own expression field, where a depth gate never sees it.
- A subject with no rdf:type has no star, so its triples vanished: the
canonical "signals per track" returned 1 for every track.
- A sub-SELECT's projection and modifiers never reach the plan, so a
subquery LIMIT 5 was ignored and the count covered everything.
- LIMIT was pushed past filters the scoper cannot express, so
FILTER(REGEX(...)) LIMIT 10 fetched ten arbitrary rows and oxigraph
filtered them down to a handful. That one is a wrong answer on the
*existing* route too, not just the pushdown.
QueryPlan now carries `exact`, computed where the loss happens: filter
extraction reports whether it understood the whole expression, triples are
checked to have a scoped subject or a traversed path, and sub-SELECTs are
detected. The analyser refuses unless the plan is exact, and `sql_limit` is
withheld unless it is.
The check is "did the planner drop anything at all", not "did I recognise
enough" — the second is what let each of these through.
Also, from the same review round:
- SolutionSpec carries `projected`. `ORDER BY DESC(COUNT(*))` creates an
aggregate with no AS name, and emitting it handed the caller a column
named 75c423f63bcf6d1e64f7265c619d5f56. Only what SELECT asks for is
emitted now; a variable projected but neither grouped nor aggregated is
refused rather than invented.
- peel's Slice arm composes instead of assigning, so LIMIT 3 around a
subquery's LIMIT 10 yields 3. Subqueries are refused as inexact, so the
nesting cannot arise today — the arithmetic is there so that stops being
load-bearing.
And a process fix behind all of this: my edit scripts printed success
whether or not a pattern matched, which is how the previous commit's
claimed limit refactor was silently lost while the tests I ran afterwards
measured unchanged code. Every anchor is asserted now, and that refactor
(pushable_limit, folding extraction and refusal into one walk) is in this
commit — the previous message overstated what it delivered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit's guard had a hole of exactly the kind it set out to
close, and the review measured it:
?s a asset360:Signal . ?s <urn:unknown> "x" -> exact -> ELIGIBLE
?s a asset360:Signal . ?s ?p "x" -> exact -> ELIGIBLE
Both counted every Signal. The guard reconstructed exactness afterwards by
inspecting triple *subjects*, so it could only catch the drop sites it knew
to look for — and Phase 1 drops triples at four others: a variable
predicate, a predicate matching no slot, an inline constant inside
OPTIONAL, and collect_values_filters, which still returned () while its
sibling had been changed to report.
Now each drop site records the fact where the knowledge already is, which
is both correct and smaller: `every_subject_scoped` and the `traversed`
out-param that existed only to feed it are gone, and walk_paths is back to
7 arguments. One exception earns its keep: an untyped subject may be a
legitimate step inside another star's nested structure, and only the path
walk knows which — so the drop site records the *variable* and the walk
clears the ones it explains, rather than the check being re-derived.
`exact: bool` becomes `Option<Inexact>`, nine variants, one per cause. The
refusal now names what was dropped and the single rewrite that fixes it;
before, IncompletePlan was the only refusal with a generic detail, no
location, and a hint listing four rewrites of which three never applied.
Also from the same review:
- sql_limit was assigned in Phase 4 and overridden in Phase 7,
reinstating the two-owner shape pushable_limit exists to prevent and
contradicting its own doc comment. One assignment now, after the causes
are known.
- The Slice composition existed in two unreachable copies whose
sum-and-min arithmetic was not the right composition anyway — an inner
LIMIT applies before an outer OFFSET. Both refuse now: pushable_limit
pushes nothing when a nested Slice is present, and the analyser blocks
the query. An unreachable path with plausible-but-wrong semantics is
worse than no path.
- `_ => CmpOp::Lte` over four bound variants is exhaustive. Widening a
prefetch was survivable; with the plan as the answer, a fifth spargebra
comparison silently becoming `<=` is a wrong number.
- `cargo run --bin stub_gen -- --check` failed at head: the .pyi had none
of the Pushdown* classes, QueryPlan.exact or path_bindings. Regenerated,
and it is an AGENTS.md pre-commit item my "all clean locally" list had
omitted.
Tests: the C1 fix had zero coverage — the line gating sql_limit on
exactness, which fixes a shipped wrong-results bug, was asserted nowhere,
and nothing read plan.exact. Four tuples added to the existing limit
table, plus one table mapping every drop site to its expected cause and
one asserting that expressible queries stay exact (including a nested path,
the legitimate untyped subject).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…f-contained
Second re-review round. Everything below was measured against the analyser
before and after, not read.
Wrong results on the live route:
- pushable_limit discarded OFFSET. `LIMIT 10 OFFSET 20` gave sql_limit=10,
so the consumer fetched ten objects and the engine then offset twenty of
them and returned nothing. The pushable value is start + length: the
fetch has to cover the whole window. Same bug class as the one this
function was written to own, inside that function.
- GRAPH and SERVICE were walked transparently with the plan still claiming
exactness, so a named-graph pattern would be answered from the default
graph and a remote pattern from local SQL. Both now name their cause.
- `?s :hasName ?v ; :asset360_uri ?v` binds one variable through two
slots, which is an equality between them. The plan carried the slots with
nothing tying them, so which binding a consumer picked decided the
answer — hash order and all.
- A scalar leaf was inserted into the path walk's `traversed` set, which
cleared the drop recorded for it, so `?nm :hasName ?x` treated a literal
as a scoped subject. Only intermediate steps clear a drop now.
- VALUES inside OPTIONAL was pushed into the fetch: the depth gate covered
Filter but not Values.
- Connectivity was checked by counting edges, so three classes with two
edges between the same two of them left the third as a cross product.
Now the join graph is walked.
The verdict is now self-contained (C2, the item that lands on the
supported subset rather than the edges). `Eligible` carries the plan it
was derived from, because the solution spec is not a description of the
query: `FILTER(?l > 5)` on a `COUNT(*)` yields a solution with no bindings
at all, and a bare aggregate's solution does not even name the class. Both
halves are needed, nothing said so, and `exact`'s own doc read as though
True licensed answering from the verdict alone. The consumer also stops
re-scoping, so there is no second parse that could disagree.
One over-refusal fixed: `FILTER(?v IN (...))` is the same constraint as
`VALUES ?v { ... }`, which was already pushed. Accepting one and refusing
the other made the subset depend on how the query happened to be written.
Tests: every item above, plus the two that had none — Container::List and
::Mapping were entirely unexercised, so nothing said which unnest a
consumer owes, and getting that wrong is a silent count error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three rounds of review each closed the previous round's drop sites and
found a new one — subjects, then predicates, then join type / partial
VALUES / a slot bound twice. That is what enumerating `continue` sites by
hand gets you, so this round changes the default instead of extending the
list.
Phase 1 now works off a set of unconsumed triple indices. Only a path that
*fully represents* a triple marks it consumed, and whatever is left over
makes the plan inexact. A drop site nobody thought of is reported rather
than silent; naming the cause is a separate, best-effort step that the
verdict does not depend on.
That alone fixes three of this round's findings, and two of them are ones
the enumeration had missed:
- One slot read through two variables (`:kinds ?x ; :kinds ?y`) was a map
overwrite: the plan described a single read where the query pairs the
slot's values with each other. oxigraph 13, plan 5. The exact mirror of
the one-variable-two-slots case the last round added — and `GROUP BY ?x`
was refused only by accident of insertion order.
- A second, different rdf:type was discarded, so `?s a :Signal ; a :Track`
counted every Signal where SPARQL returns none — and the answer depended
on which type came first.
- A constant object carrying a language tag or datatype lost it:
`:name "BX1"@en` became `= "BX1"`, matching rows the query excludes.
Fixed alongside:
- VALUES with UNDEF. `Some(Some(term))` skipped the UNDEF cell, but UNDEF
is *no constraint* — dropping it turned a union into an intersection, so
`VALUES ?nm { "BX1" UNDEF }` gave 1 where oxigraph gives 3.
- An OPTIONAL reference between two mandatory classes. The guard tested
star.is_optional and never the edge's join type, so a Left edge both
satisfied the connectivity check and hid that the classes are unrelated:
2 Signals × 2 Tracks is 4 solutions, and rendering the edge as a join
answers 2.
- C7. A required and an optional nested path produced byte-identical
bindings and have different answers, so PathBinding now carries
optionality and the optional form is refused until the plan can express
it.
Docs both reviewers flagged: sql_limit means OFFSET + LIMIT, in the field
and the Python getter, and inexact_reason lists all nineteen causes with a
note to treat an unknown one as inexact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nner Review cleanup, no behaviour change: 207 tests pass unchanged before and after each step. * One `ScopeError` -> `PyValueError` conversion via `Display` instead of two fourteen-arm match blocks fifty lines apart, which stayed identical by luck. * `term_descriptor` and `containers_along` walked the same slot path twice, one for the term shape and one for the container modes; they are now one `resolve_column` returning both, and `containers_along` is gone. * `binding_for` resolved a variable once to decide which star owns it and again to find where the value sits. They are the same question: one `find_map` answers it, the subject special case falls out of it (the empty path), and the second refusal that followed was unreachable. * `TermRef` held a class IRI and a copy of the slot path the binding already carried; the class IRI moves onto `BindingSpec` and the per-binding `Vec<String>` clone goes with the struct. * Four linear `slots()` scans become the class's own name index, or -- in the schema-wide candidate scan, where only existence is asked -- a `HashSet` of names built once per class rather than rescanned per slot. * One `From<sparql_scoper::QueryPlan>` for the Python wrapper, built by hand at both entry points before, and `exact` derived from `inexact_reason` rather than stored beside it: two fields that must agree are one field. * `Slice` moves into `blocks_limit_push` (was `is_holistic`), which absorbs `contains_slice`; `QueryPlan.__repr__` prints the exactness it was hiding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011kG4omRNzQDAHGHvGSTYeX
…he reachability walk `Inexact::OptionalJoin` and `Inexact::OptionalPath` were declared, documented, and listed among the values `QueryPlan.inexact_reason` can return, with no construction site anywhere: both shapes are refused inside `analyse_pushdown` as `BlockedCode` instead. A consumer branching on the documented set was handling two strings it could never see. They are gone rather than wired up, because neither belongs to the scoper's question. An OPTIONAL reference between two classes is something the scoper describes correctly for a prefetch — it builds the LeftJoin tree for exactly that — and only rendering it as a SQL join implies a relationship the query does not require, which is a pushdown concern and already refused there. What the plan *was* hiding is the optionality of a nested path: `PathBinding` has carried `optional` since the C7 fix, and the Python conversion dropped it, so `path_bindings` described a required and an optional read identically while `exact` said the plan was complete. It is now the third element of the tuple. Also here, all no-ops for behaviour: * `stars_reachable_from` replaces the two hand-written copies of the same undirected fixpoint — Phase 2b's disconnected-OPTIONAL check and the planner's `disconnected_star` — so "connected" cannot come to mean two things. * the `drop` closure becomes `record_loss`. It shadowed `std::mem::drop`, and renaming only the literal call sites turned three `drop(cause)` calls into silent discards that still compiled: five tests caught it, which is the argument for the rename. * `term_descriptor` and `CmpOp::as_sql` had no callers outside tests; the former is now a test helper, the latter is gone. * the three tables asserting a drop site records its cause are one table, with the blank-node property list (`UnscopedSubject`) that none of them covered. * `every_inexact_cause_is_fully_described` enumerates the causes, so a variant added without its three strings — or without its line in the Python docstring — fails to compile rather than shipping half-described. Tests 207 -> 206 (two folded, one added); clippy, fmt and stub_gen --check clean.
…e the working set
Five wrong numbers, each measured against oxigraph, each with `exact = True`.
**A claim was never revoked.** `unconsumed` is cleared by whichever arm
represents a triple, but a claim is only as good as the star it was made
against — and a star whose class does not resolve is discarded whole. Its
triples were then represented by nothing while still counted as claimed. The
path walk is what usually rescues them, so a `StarBuilder` now remembers the
triples it claimed and hands them back unless the walk really did carry every
one: a constant on a nested step is not carried (a path says where a value
lives, not what it equals), and neither is an `rdf:type` on it. Three shapes
that were eligible and wrong:
?s a Signal ; :documents ?d . ?d :title ?ti ; :docId "D1" (oxigraph 1, plan 2)
?s a Signal ; :location ?c . ?c a <NotAClass> ; :longitude ?lo (oxigraph 0, plan 2)
?s a Signal ; :location ?c . ?c :longitude ?lo ; :name ?x (oxigraph 0, plan 2)
The third is the reason `get_slot_by_uri` is not enough on its own: it is
schema-global, so Phase 1 claims `:name` on `?c` although `Coordinates` has no
such slot. Relatedly, a subject can be scoped by *which slots it uses*, and
that fallback answers even when the stated `rdf:type` names something the
schema does not have — so the resolved class must now equal the stated one.
**The working set covers triples, and three losses are not triples.**
* A multi-column `VALUES` lists tuples. `("BX1" 4) ("BX2" 3)` admits two
combinations; one `IN` per column admits four.
* A language tag or a datatype never survives a text comparison, and only the
inline-object site checked for it. `FILTER(?nm = "BX1"@en)` and
`VALUES ?nm { "BX1"@en }` both pushed `= "BX1"`, matching a row oxigraph
excludes. All three sites now ask the *column* how its values render, via the
same descriptor the renderer uses — so a numeric column still takes `> 5`,
which is what makes comparison pushdown work at all.
* One multivalued slot read as a constant *and* through a variable is the same
self-join as two variables, in the other direction, and only the
two-variable case was caught.
**And one that is not a loss but a multiplicity.** A multivalued slot read
through a variable nothing groups or aggregates still multiplies solutions:
`?s a Signal ; :trafficKinds ?k` with `COUNT(*)` counts one solution per kind,
where a plan that never mentions `?k` counts records. `containers` is how a
consumer reproduces that, and a variable with no binding has no container and
no instruction — so the planner refuses. Grouping by it stays eligible, since
the multiplicity is then the answer.
Not changed, deliberately: `:length "3"` stays pushable and `:length 3` stays
refused. The review suggested that pairing was inverted, but `literal_and_type`
in the pinned rust-linkml-core emits `new_simple_literal` for an integer range
with no custom `rdf_datatype_iri`, so the stored term *is* the plain literal —
which is also what `describe_slot` reports, and the differential oracle pins
the two together.
207 tests; clippy, fmt and stub_gen --check clean.
…op revoking walked paths Review found three regressions in the previous commit, all measured against oxigraph. Two of them I introduced. **A constant is pushable when it is the same RDF term the column renders.** `push_form` treated "the descriptor names a datatype" as "the stored text is not the term", so it reported `Tagged` for every properly declared type and the numeric branch of `literal_pushable` could only fire when a schema's `linkml:types` was *not* loaded — which is precisely where a typed numeric constant matches nothing. Inverted, in both directions: * a schema that resolves `integer` to `xsd:integer` stores `3`, and every numeric filter, `IN` and `VALUES` pushdown on it was refused — a regression against ff68af7, each refusal also withholding LIMIT pushdown; * a schema that leaves the type unresolved stores the plain literal `"3"`, and there `FILTER(?l > 2)` was pushed while oxigraph answers nothing. So the rule now asks the column: a constant is pushable iff its language and datatype are the column's, with a plain literal's `xsd:string` and a `None` descriptor treated as the same term. `numeric` goes back to being about whether an ordering is meaningful, which is all it ever described, and the twelve hardcoded XSD IRIs are gone with it — the question was never about the query's literal in isolation. The scoper's fixture had no `types:` block, which is why nothing caught this and why two of its tests asserted the broken behaviour. It now declares them, as `tests/data/asset360.yaml` does through its `./types` import, so the tests run the configuration production runs. **A walked path is not an unrepresented one.** The revocation check required every value variable of a discarded star to come back as a path binding, but `object_variables` holds intermediate steps as well as leaves, and an intermediate node is deliberately never a path binding — it serialises as a blank node, so the walk records it as a step and continues. Any path past two hops was therefore refused although the plan carried the whole chain. Accepting `traversed` alongside `path_bindings` fixes it; the fixture gained the three-hop inline chain it never had. **The multiplicity guard asked the wrong function.** `resolve_column` declines to describe a multivalued *inlined* range at all, so `documents` read as single-valued and `COUNT(*)` over it counted records where SPARQL counts documents. It now asks the slot's container mode directly, and covers a multiplying *hop* inside a path as well as a slot read off a star — while skipping a step that some bound value already extends, since that value's `containers` describe every hop of its own path. Also, all verified by execution: * the stated-type-equals-resolved-class check added last commit is deleted. It could never fire: `resolve_star_class` returns the stated IRI verbatim as the class URI, and the slot-inference fallback is only reachable when no type was stated. Its justifying comment misdescribed that function, and the shape it claimed to catch is caught by the discarded-star path. * `DiscardedStar` was a field-by-field copy of `StarBuilder`; the borrow does. * `FILTER(?ref = <iri>)` now pushes, like `FILTER(?ref IN (<iri>))` already did. One rule in two arms, and the last commit widened the gap by teaching only `IN`. 210 tests; clippy, fmt and stub_gen --check clean. CI's `rust` job gates this branch and was green on the previous head.
…filter is a containment test C9, both halves. Each was eligible on the pushdown route with the fixture this repo already ships, and each answered a question the query did not ask. **An inlined slot never joins.** A `JoinEdge` says the slot holds the other class's URI, and Phase 2 emitted one for any object variable that happened to be a star — never consulting `SlotInlineMode`, though `walk_paths` and `describe_slot` both do. Give a nested structure its own `rdf:type` and the correct path binding was *replaced* by an edge whose documented rendering, `t1.object_data->>'location' = t0.asset360_uri`, names a column that cannot exist: `Coordinates` is `inlined: true` with no identifier. It was ELIGIBLE. An edge is now emitted only for a slot whose inline mode is `Reference`; anything else records `TypedNestedStructure`. The star for the nested subject is still built — Phase 1 makes one for any typed subject — but nothing joins to it, so it costs a fetch that returns no rows rather than a wrong number. The hint says to drop the `rdf:type`, and that is worth spelling out because it is a real fix rather than a deflection: the writer *does* emit `_:b rdf:type asset360:Coordinates` for an inlined object, so the typed form is a legal question, and I checked through the writer into oxigraph that it answers exactly what the untyped form answers. The untyped form is a path, exact and eligible. This also subsumes the resolvable-but-wrong-class shape (`?c a asset360:Track`), which produced the same bogus edge. **A filter on a multivalued slot is a containment test.** `Star.filters` had no container information and its documented rendering is `object_data->>'field' = 'value'`, which compares an array's own text: for `?s a Signal ; :trafficKinds "m"` oxigraph answers 1 and that SQL answers 0. `Star` now carries `multivalued_fields`, so a consumer can render the `jsonb_array_elements_text` containment test instead — and the same list answers the other question a value on such a slot raises, which is that a record with three values answers a SPARQL question three times. Additive on the Python surface: `filters` keeps its shape and points at the new attribute. 212 tests; clippy, fmt and stub_gen --check clean.
…stop calling an ordinary query an OPTIONAL
`typed_nested_structure` shipped missing from the list the guard test reads, from
`QueryPlan.inexact_reason`'s docstring, and from the generated `.pyi` — the
published contract. The commit that added the guard claimed a cause without its
strings would "fail to compile"; that was wrong. `as_str`, `detail` and
`instead` are exhaustive matches and the compiler does enforce those, but
`const ALL: [Inexact; 19]` is a plain array literal and went on compiling when a
twentieth variant arrived, so the test built on it kept passing while the cause
was absent from it and from the docstring beside it.
Both restatements are now derived:
* the enum is declared through `inexact_variants!`, which emits `Inexact::ALL`
from the same rows — the list cannot fall behind the enum it describes. The
three string tables stay hand-written, because three of the `instead` hints
are deliberately shared between two causes each and a row-per-cause table
would duplicate those literals, which is the hazard this is meant to remove.
Verified string-for-string identical across all twenty causes before and
after.
* `sparql_inexact_reasons()` serves the set to Python, and the getter's
docstring points at it rather than listing it.
Also, N8: the plainest query there is —
`SELECT ?nm WHERE { <.../signal/A> asset360:name ?nm }` — was **rejected**
with "OPTIONAL block introduces ?_const_subject_0 … disconnected OPTIONAL is not
supported yet", about a construct the query does not contain. A constant-IRI
subject names its instance instead of typing it, so `type_depth` keeps its
`usize::MAX` sentinel, and `type_depth > 0` read that as "introduced inside an
OPTIONAL". Where no type was stated the star is now as optional as the
shallowest triple that mentions it; a stated type keeps the old rule, so
`OPTIONAL { ?s a :Signal }` still makes its star optional.
213 tests; clippy, fmt and stub_gen --check clean.
…he star a carried path belongs to
Four wrong answers, all with `inexact = None`, all measured against oxigraph
through this system's own writer.
**The BGP inline-constant arm never asked the column.** `?s :length "3"` was
pushed as `Eq("3")` on a column storing `3`, and `?s :length 3` — the form that
matches — was refused; `:description "hello"` matched a record oxigraph excludes
on a slot with `in_language: en`; an IRI constant was pushed against a literal
column and a literal against a `uriorcurie` one. The FILTER and VALUES routes got
all of these right, because the rule was rewritten there and not here. The
previous commit's message said "one rule in two arms"; there were four, and this
is the one that stayed wrong.
It now builds the column's `PushForm` from the `SlotView` it already holds and
routes literals through `literal_pushable`, IRIs through `PushForm::Iri`. An
identity slot bypasses the rule deliberately: the writer emits no triple for it,
and a constant there is hoisted to an indexed `asset360_uri` lookup rather than a
JSONB text compare, so the term question does not arise.
**A datatype match is not SPARQL `=`.** On a numeric column SPARQL compares
values while the pushed condition compares text, so `= "003"^^xsd:integer`
selected a record `object_data->>'length' = '003'` never finds — a wrong answer
on the deployed prefetch route, not just the aggregate one. A numeric constant is
now pushed only in the form the stored text is written in.
**The language branch was dead by construction.** A language-tagged literal's
datatype is `rdf:langString`, which never equals a column's, so testing the
datatype first refused *every* constant on a language-tagged column, including
the right one. Language is now decided first.
**A carried path was matched by spelling, not by star.** `Signal.documents` and
`Track.documents` are both legal, and a binding on one excused an unaccounted
multiplication on the other: two Signal documents and three Track documents
answered 6 in oxigraph and 3 from the spec. The predicate now requires the
binding to belong to the same star. `Track` gained a same-named `documents` slot
so the fixture can tell the difference.
**And an oracle, instead of a seventh hand-written case.** Six bugs here shared
one shape — a rule applied at three sites out of four, or to one operator and not
its twin — and each was found by hand. `pushed_filters_match_sparql` sweeps every
column kind against every way of writing a constant and asks the only question
that matters: when the plan says exact, does rendering its filter the documented
way give oxigraph's answer? Its second half asks the converse, because an oracle
that treats refusals as safe cannot see the feature quietly refusing everything —
which is exactly how the language branch died unnoticed.
I verified the oracle fails on each bug above before the fix: reverting the
inline arm trips it on `:length "3"`, reverting the canonical-form check trips it
on `"003"^^xsd:integer`, reverting the language order trips the second half on
`"hello"@en`, and reverting the star check trips the multiplicity test.
215 tests; clippy, fmt and stub_gen --check clean.
`FilterCondition::Cmp`'s doc says the slot's term descriptor decides whether a comparison casts -- but a slot that appears only in a `FILTER` is in no binding, so no descriptor ever reaches the consumer. It compared as text, and text does not order the way a number does. Two wrong answers on this repo's own fixture, both silent, in opposite directions: * `FILTER(?year >= 10) GROUP BY ?name` returns three groups where there are two, because `'9' >= '10'` is true as text -- the aggregate route answers this itself, so the wrong number is final; * `FILTER(?year > 9)` with SUM/AVG/MIN/MAX fetches nothing at all, because `'2001' > '9'` is false as text, and the engine then aggregates an empty graph into `SUM = 0` with MIN and MAX unbound. A pushed comparison that *narrows* is the one thing the over-fetch contract does not survive. `Star::numeric_fields` answers it, in the same shape as `multivalued_fields` from the previous commit and for the same reason: asked of the schema once, at the point where the star is built, because the consumer cannot derive it and getting it wrong is invisible. It goes through `resolve_column`, so a filter and a group key on one slot cannot disagree about its type -- there is one term rule, and this is a fourth caller of it rather than a second list. Additive on the Python surface; `filters` keeps its shape and its docstring now points at both lists. 216 tests; clippy, fmt and stub_gen --check clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011kG4omRNzQDAHGHvGSTYeX
…ed code `PushForm::Tagged` refused every constant on an enum-valued slot, so on the review dataset "how many signals of type KSS" -- the plainest filter a report asks -- could not push down at all. It fell back to the engine, which fetched 23505 signals, built 1.5M triples and returned `limit_exceeded` after 30 seconds. The refusal had the right instinct and the wrong conclusion. The stored text is a permissible value; the term it renders as is that value's `meaning` IRI when it has one, and the plain literal otherwise. So a constant is not compared against the column, it is translated *backwards* -- from the term the query wrote to the code the column stores. The real `signalType` is why both halves matter: `GSA` carries a meaning and renders as an EULYNX IRI, while `KSS` carries none and renders as itself, in one enum over one column. * an IRI constant selects the codes whose meaning it is -- a list, because two codes may share one meaning; * a plain literal selects itself, which also covers a value the data holds that the enum does not declare; * the literal spelling of a *mapped* code selects nothing, and that is recorded as a loss rather than pushed: `= "GSA"` as text would answer with 12072 records where SPARQL answers with none. `constant_texts` is the one rule, used by the inline arm, `FILTER(?k = ...)` and `IN`. Those three were copies before and the copies drifted -- only `IN` had learned that a reference column compares against an IRI -- so this adds a fourth caller rather than a fourth copy. Comparisons are deliberately not included: an ordering over codes is not an ordering over the terms they render as. The fixture gained a partially mapped enum, shaped like the real one, and `FilterCondition` gained `PartialEq` so a test can state the condition it expects rather than its debug string. 218 tests; clippy, fmt and stub_gen --check clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011kG4omRNzQDAHGHvGSTYeX
Grouping already reached into JSON -- `?m :districtName ?d` groups two slots down -- but filtering did not, and drill-down is filter *and* group. On the review dataset every filtered question refused: the constant form (`?m :zoneName "Charleroi"`) as an unrepresented triple, the FILTER form (`FILTER(?z = "Charleroi")`) as an expression that could not be turned into a condition. Both fell back to the engine, which fetched 23505 signals and returned `limit_exceeded` after 30 seconds. Neither refusal was about the condition. It was about the key: a filter was addressed by slot name, and a value two slots down has no slot name. So conditions are now keyed by *path*, `Star::path_filters` carries the ones longer than a single slot, and the two ways of writing one arrive as one condition rather than two shapes of one. Three things had to move for that: * the path walk now runs before the filters are collected, so a nested value has a path to be attached to when the FILTER is read. It only ever needed the stars, which exist by then; * `ValueColumns` and the filter map are keyed by `Vec<String>`, because `maintenanceUnit.zoneName` and a top-level `zoneName` are different columns that one flat key would merge; * a constant on a nested step is now *carried* by the walk, so the discarded star that held it counts as represented. That check was the one place saying a path records where a value lives and not what it equals -- true until this commit, and the reason the constant form was refused. Two guards, both silent if skipped, in `path_push_form` so they cannot drift apart: every hop has to be single-valued, or the condition is a containment test over elements rather than an equality; and no hop may be optional, or pushing it drops the rows a LEFT JOIN exists to keep. Deliberately not covered: a constant on a nested step whose slot is an enum. Phase 1 records the constant against the slot's own term rule, and that is enough for every other range, but an enum needs the term as written to translate it back to a stored code. The FILTER form of the same question is translated correctly. Contract change for consumers: a consumer that renders `filters` and not `path_filters` now asks a weaker question than the query did, which on an aggregate route is a larger number with no warning. The getter's docstring says so. Wheel and app ship together. 220 tests; clippy, fmt and stub_gen clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011kG4omRNzQDAHGHvGSTYeX
`peel` walks the modifiers above the grouping and never handled `GraphPattern::Filter`, so a `HAVING` -- which is a filter wrapping the `Group` -- hid the `Group` underneath it. The verdict was `not_applicable`, the same answer a query with no aggregate at all gets. That was the worst outcome available. `blocked` carries a reason a caller can act on and, since the consumer refuses early on a class it cannot materialise, costs nothing; `not_applicable` means "nothing to see here", so the query went to the engine, which on the review dataset spends thirty-two seconds building 1.5M triples before reporting a triple limit. Measured, on `GROUP BY ?regime HAVING (COUNT(*) > 1000)` over 23505 signals. Peeling stops at the `Group`, so any filter it reaches is above the grouping and cannot be a WHERE clause `FILTER` -- those live inside the group's own inner pattern, which this never descends into. Refused rather than translated for now. SQL has `HAVING` and it is the natural next step, which makes this a missing feature rather than a shape that needs rewriting; the detail says as much. 221 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011kG4omRNzQDAHGHvGSTYeX
…edger
The first half of making the plan the program. This adds the artifact and
its invariant; nothing emits one yet, so nothing changes behaviour.
Pushdown has been a *verdict*: the planner said eligible or blocked and
the caller chose one of two whole-query routes. That put the query's
meaning in the caller, where it grew a routing branch, a
fallback-viability heuristic, a feature-detection flag and a 422 -- sixty
lines of the executor guessing at what the planner already knew.
And a plan made of optional lists can be under-read. A consumer built
before `Star::path_filters` existed ignored it, answered 21956 where the
answer was 4108, and called the plan exact while doing so. That is the
failure this design is shaped around.
An `ExecutionPlan` is an ordered set of `Pass`es -- `Sql` or `Engine` --
and the `Obligation`s each one discharges. An obligation is one thing the
query asks for: a triple pattern, a filter, a grouping, an aggregate, an
ordering, a slice. Two rules make the artifact trustworthy:
* **The ledger balances.** Every obligation appears exactly once, in a
pass or in the residual. `ledger_balances()` says so mechanically, and
`residual.is_empty()` is what "the plan describes the whole query" used
to mean as a flag beside the plan. The direction of the default is
inherited and must not be inverted: an obligation is residual *unless*
a pass claims it, the same inversion that turned four silent drop sites
into loud ones.
* **The contract is versioned**, so an executor can refuse a plan it does
not fully understand instead of running the part it recognises.
`Display` prints it, completely -- an obligation absent from the string
is absent from the plan:
ExecutionPlan (contract 1, exact)
pass 0 SQL asset360:Signal → ?kind ?n
scan asset360:Signal as ?s
discharges o0 o1 o2 o3
residual (empty)
obligations
o0 type ?s a asset360:Signal
o1 triple ?s asset360:kind ?kind
o2 group GROUP BY ?kind
o3 aggregate COUNT(*) AS ?n
Written for two readers. Someone debugging sees which filters reached SQL
and which did not, with the term shape beside them -- `enum→iri`,
`xsd:integer`, numeric -- because that is what decides an answer and none
of it is visible in the query. And a reviewer can read a corpus of plans
instead of the planner: every `oN` appears once, so the ledger checks by
eye.
Printing it immediately showed two things worth fixing, which is the
argument for having it. Aggregates read as spargebra's internal variable
(`?bd1a8feb2eb7c1b061520830b70d6d3a`) rather than the name the author
wrote, since `(COUNT(*) AS ?n)` is a `Group` plus an `Extend` one level
up; and full IRIs made the lines unscannable. Both fixed here.
Supporting `Display` impls for `FilterCondition`, `OrderTerm`,
`Measure::render` and `TermDescriptor::shape` come with it -- each exists
so the plan can show the thing that decides an answer rather than a debug
dump.
224 tests; clippy, fmt clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011kG4omRNzQDAHGHvGSTYeX
The second half: the planner now emits the artifact, and the residual is
the working set it was already computing.
`scope_parsed` kept `unconsumed` -- the triples nothing represented --
and threw it away, keeping only the first cause as `Option<Inexact>`.
That answered "was anything dropped" and lost "*what* was dropped", which
is the difference between refusing a query and handing the rest to
another pass. It is now a field, and it is where `residual` comes from:
no new bookkeeping, the same default-deny working set, exposed.
`plan_query` parses once, scopes once, analyses once. Two shapes come out:
ExecutionPlan (contract 1, all in SQL)
pass 0 SQL asset360:Signal → ?kind ?n
scan asset360:Signal as ?s
discharges o0 o1 o2 o3
residual (empty)
ExecutionPlan (contract 1, SQL narrows, engine finishes)
pass 0 SQL asset360:Signal
scan asset360:Signal as ?s
discharges o0 o1
pass 1 ENGINE input [0]
because filter_expression — a FILTER this cannot turn into a
SQL condition was left for the SPARQL engine
discharges o2 o3 o4
residual (empty)
The second shape is what the fallback route has always done -- fetch a
superset, let the engine apply the real query -- said out loud. The scan
still claims the triples it represented and the engine claims the rest,
including the grouping, because a scan-only pass did not group and must
not say it did.
Printing it corrected the header I wrote in the previous commit. `exact`
was `residual.is_empty()`, and with an engine pass claiming the leftovers
that is true of every plan, which made the word meaningless. The question
a reader is actually asking is `sql_only()`: are all the passes SQL, so
the answer needs no objects materialised. A non-empty residual now means
something sharper -- an obligation with *no* pass at all, which is a
planner bug or a consumer that has no engine to fall back on. That state
is exactly what a stored minibi question needs to refuse on, so it earns
its place rather than being an error case.
`analyse_pushdown_scoped` takes a decomposition the caller already has.
The endpoint calls `sparql_scope` and then `sparql_pushdown`, which
scopes again: every request does the work twice and ends with two results
that could in principle disagree. `869b89443` fixed the disagreement by
shipping the plan inside the verdict; this removes the second call.
`sparql_scope` and `sparql_pushdown` stay for now. The deployed app and
star-data call them, and they live in another repo -- migrating them is
the consolidator MR, and doing both at once would mean the wheel and the
app could only ever be released together. Given a version skew produced a
wrong number last week, that transition should be deliberate.
227 tests; clippy, fmt clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011kG4omRNzQDAHGHvGSTYeX
plan_query() existed but nothing outside Rust could call it, so the Django side still made two calls that each re-parsed the query and answered a question the artifact already contains. Exposes plan_query plus ExecutionPlan and PlanPass. The accessors are the ones a consumer actually branches on — is_accounted before running anything (a residual means the plan answers a different question than the one asked), sql_only for a caller with no fallback, and per pass the kind, what it emits, what it discharges, its QueryPlan and solution, or the causes that forced an engine pass. __str__ delegates to the Display already written for the ledger, so a plan can be printed in a shell, a log line or a failing test and read as prose: the state, each pass with what it discharges, the causes, the residual, and every obligation the query imposed. Both kind vocabularies are closed. A consumer meeting a pass kind it does not know must refuse the plan rather than skip the pass, which would silently answer something else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An engine pass records causes, which explain a slower answer. They do not say which construct put an aggregate outside the subset, nor what shape would work instead — and that is what the endpoint sends back when the fallback cannot answer either. Recovering it took a second analyse call that re-parsed the query, which is the duplication this refactor exists to remove. So the artifact carries the refusal: code, detail, optional location, and the rewrite hint, reachable from Python as .blocked and rendered in the ledger under "not pushable". A refusal is not a broken plan. The engine pass still answers the query, so is_accounted stays true and the endpoint is free to serve it — the refusal only explains why SQL could not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kervel
changed the title
…> plan
The ledger already gave a rewrite something to check itself against: every
obligation discharged exactly once. What it did not give was something
local to rewrite. An SQL pass carried two composite structures shaped for
rendering -- the star decomposition and, when it groups, the whole
solution spec -- so pushing a filter out of the engine meant reaching into
a HashMap of slot conditions, moving an obligation id, and remembering to
recompute the engine pass's causes so they did not lie. Every rule would
redo that surgery and nothing would check it.
SqlPass.ops states the same pass as nodes: scan, filter, join, unnest,
group, sort, distinct, slice, project. Inputs are indices, so the tree is
a flat vector and inserting or moving a node is an edit rather than a
rebuild, and each node carries the obligations it discharges -- which is
what makes OpTree::claims() checkable against the pass, and the ledger the
postcondition of any rule.
Also puts one distinction in the data that was previously only a
convention. When an engine pass decides the answer, SQL may apply a filter
to narrow rows while the engine stays authoritative for SPARQL's term
semantics; when SQL answers alone, the same filter *is* the answer. That
difference was only inferable by comparing which pass claimed the
obligation. Enforcement::{Enforces, Narrows} says it, so a pushdown rule
can ask a node whether removing it would change the answer.
Lowered from plan and solution, so it says nothing they do not, and the
tests hold it to that: claims equal the pass's claims, inputs precede
their node, a join lowers to a join over two scans, and a grouping pass
lowers in execution order. The renderer migrates onto this next, and the
two rendering-shaped fields go.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The operator tree existed but nothing outside Rust could read it, so the renderer still worked from the two rendering-shaped views of the same pass. PlanOp exposes one operator: kind first, then only the accessors that kind has -- class and existence checks for a scan, path and condition and enforcement for a filter, the foreign-key slot for a join, bindings and keys and measures for a group. Other kinds return None or an empty list, so a renderer branches on kind and reads what belongs to it. Inputs always precede their operator, so the list can be walked front to back. `enforcement` is the accessor that makes a pushdown rule possible on the Python side too: a "narrows" filter can be dropped and the answer is still right, only slower, while dropping an "enforces" one answers a different question. Read what a grouped query lowers to: 0: scan inputs=[] star=?s TunnelComplex 1: filter inputs=[0] star=?s hasName enforces 2: unnest inputs=[1] star=?s hasTrafficKind 3: group inputs=[2] keys=[0] measures=['count'] discharges=[0..7] 4: sort inputs=[3] 5: slice inputs=[4] limit=10 6: project inputs=[5] vars=['kind', 'n'] Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Starting the renderer migration surfaced two facts the operators dropped, both of which are wrong answers rather than errors if a renderer cannot see them: - a filter's comparison being numeric. It cannot be recovered from the path: the same slot name on two classes may differ, and a value inside a structure is not in the record's own slot list. Comparing a number as text is what makes '9' >= '10' true, which is the reason the scoper tracks numeric_fields and PathFilter::numeric at all. - a star being optional. It decides the join type above the scan and whether the scan's own conditions must tolerate a row the join did not match. Rendered as required, an OPTIONAL block drops rows the query keeps. Both now live on the node and cross to Python, with a test that lowering preserves them: a comparison on an integer slot is numeric, and a query with an OPTIONAL block lowers to one optional scan and one that is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A scan feeding the engine could be limited -- the scoper works out when a row bound is safe to apply -- but that bound lived only on the query plan, so a consumer reading operators alone would fetch unbounded where the scoper had said it need not. Emitted as a slice node that claims nothing, which is the honest shape: it narrows the fetch, while the engine still applies the query's own LIMIT and OFFSET. Dropping it costs speed, not correctness, and that is exactly the enforce-versus-narrow distinction the tree already carries for filters. With this the fetch pass is fully described by its nodes, which is what the object-fetch path needs before it can stop reading the star decomposition. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SqlPass carried three descriptions of one pass: the star decomposition, the solution spec when it grouped, and the operators. Every consumer now reads the operators, so the other two are gone -- and three descriptions of one thing is how a reader comes to use the stale one. That failure is in this module's own docstring: a consumer that had not heard of path_filters answered 21956 where the answer was 4108, and called the plan exact while doing so. The printout reads the nodes too, so it shows the tree that runs rather than a structure beside it. It gained two things worth having in an explain output: a filter says whether it enforces its obligation or only narrows the rows, and a scan says whether its star is optional. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pushdown is decided once per query today, so one filter SQL cannot express costs the whole grouping. A planner that starts from "the engine does everything" and moves work down a step at a time is partial by construction -- but it needs a starting point, and the operator set in sparql_ops.rs cannot express one: no expression tree, no nodes for union/minus/path/bind/sub-select, no per-node executor. So: sparql_refine.rs, a second artifact holding the query faithfully with every node engine-tagged. Its obligations are obligations_of's, unchanged, each claimed by the node that came from the same piece of algebra, and its residual is empty -- a naive plan is already correct, being what the endpoint does when nothing is pushed. Nothing here is reachable from plan_query. Five invariants, not four. 28d's fourth is the root's output kind; the fifth is the fold rule's cardinality precondition, which the design says no invariant can catch. That is true of a scan whose slots are bare names -- carry multivaluedness on the slot and the missing unnest becomes local and checkable. The plan is checked before naive_plan returns it, in every build, because the builder's one assumption is that obligations_of walks the algebra in the order it does; a check turns a broken assumption into an error rather than a plan that answers a different question. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A rule is a function on a plan: match a shape, edit nodes, move claims. The driver applies them in a fixed order to fixpoint and re-checks every invariant after each single application, so a rule that forgets to move an obligation or reparents a node without renumbering fails at the rule rather than in a result. The first rule is the type scope, chosen because its result is already known -- it is what the star decomposition produces -- so what is being proved is the machinery. It is also the one rule that can change an answer while every invariant of 28d holds: a match on a multivalued slot fans out, a scan yields one row per record, and folding one without its unnest makes a record with three traffic kinds count once. scan_with_fanout is the only way the rule builds a scan and derives the unnests from the same slot list, and the fifth invariant catches it if a future rule builds one by hand. Four more preconditions, each stated on the rule rather than left to be rediscovered: only matches joined by plain joins fold (an OPTIONAL folded in as an existence check drops the rows it exists to keep), only variable objects (a constant object is a filter, a different rule with a different claim to move), never the same array twice (a cross product is not one unnest), never two rdf:types on one subject (an intersection is not a scan of one class). One thing the design does not mention and that folding two stars found: the left-deep chain of a naive plan leaves two joins over the same pair once both stars have folded, and re-joining a fanned-out row set with itself squares the multiplicity of a repeated array value. A join whose one side the other has already joined in is dropped, which is the same idempotence the collapse of a join-with-itself relies on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…things
Two decisions about what an obligation *is*, both of which make the ledger
finer. plan_query's answers are unchanged -- the same passes claim the same
work -- but a query now raises more ids, and that is the point of the ledger.
**Per top-level conjunct.** spargebra conjoins `FILTER(a) FILTER(b)` into one
`Filter { And(..) }`, so accounting per FILTER-as-written let syntax decide the
ledger: the same question raised one obligation or two depending on how it was
typed. Worse, it made partial filter pushdown unrepresentable -- pushing `a`
and leaving `b` would have meant splitting one claim between an SQL node and an
engine node, which "discharged exactly once" forbids. Per conjunct, the claim
moves with the node that applies it, and 28d's worked example works as written.
`Or` does not split: neither half of `a || b` constrains anything alone. This
adds no obligation *kind*, so PLAN_CONTRACT does not move for it.
**Two constraints could be lost while the ledger still balanced.** The
condition spargebra lifts out of `OPTIONAL { ... FILTER(x) }` and a `VALUES`
block had no obligation at all, so a plan that dropped either balanced anyway
-- which undermines the one property the ledger exists to provide. Both are
enumerated now. VALUES gets its own kind rather than being modelled as a
filter, because a VALUES that binds a variable nothing else binds adds rows and
columns rather than constraining existing ones, and a consumer told it was a
filter could apply it as a WHERE and answer a narrower question. That is a new
kind, so PLAN_CONTRACT moves to 2; no consumer branches on it yet, so the bump
is the marker the next one checks rather than a live gate.
The naive builder follows: one Filter node per conjunct, chained, each claiming
its own obligation; the lifted condition claimed by the LeftJoin node, since it
decides whether the optional side matched and is not a filter above the join;
the VALUES obligation claimed by the inline table itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The per-application check stays a debug assertion -- it is the one that says *which* rule, and a rule breaking an invariant is a programming mistake with no sensible recovery. But it left a release-only gap: refine() returned whatever the rules produced. "Nothing executes these plans yet" is true today and is exactly the assumption that stops holding once a renderer reads them, by which point the gap is invisible. So refine() checks its result in every build and returns Result<RefineLog, RuleFailure>, the failure carrying the log because the rules that fired are the suspects and a release build has no per-step check to name one. Missing the fixpoint is not an error: every plan in the chain is correct, just less refined. It stays an assertion. The test uses a rule that edits the plan and returns false, which is the one shape the per-application check cannot see -- it only looks after a rule that said it changed something -- and is a plausible bug rather than a contrivance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n in A `Filter` whose expression renders as SQL conditions over the slots an `Sql` scan below it binds becomes `Sql`. Two things the rule has to do that the expression cannot: * rewrite `Expr::Var` into `Expr::Slot`. A naive filter compares variables, and a variable is not a column, so `to_sql` declines every naive filter by construction. Which slot binds `?name` is a fact about the scan below, and the rewrite is committed only with the flip -- an engine node keeps the expression the query wrote. * sink a pushable conjunct below an unpushable one. Obligations are per top-level conjunct and the builder chains one `Filter` each, first nearest the input, so `FILTER(?nm > "A") FILTER(REGEX(..))` pushes the comparison and `FILTER(REGEX(..)) FILTER(?nm > "A")` -- the same query -- pushes nothing, the pushable node sitting above an engine one where the frontier-is-a-cut invariant forbids it. Filters commute: they are row-preserving, bind nothing, and an expression that errors simply does not select the solution. The walk stops at any non-filter (a `BIND` below is the case that matters) and at a filter more than one node reads. Two preconditions are narrower than the star decomposition's, and both are cases where a pushed condition is not the narrowing 28d assumes: * a constant the column's values never spell selects *nothing*, and the engine leg then re-runs the query over no instances -- a wrong answer, not a narrowing. Gated with the scoper's own `push_form` / `literal_pushable`, which is why those are now `pub(crate)`. Enum columns decline rather than translate: selecting the codes that render as the term rewrites the condition, and `to_sql` has no schema. * a multivalued slot's `(star, slot_path)` means the array below its unnest and one element above it, so a condition carrying it is a containment test to one reader and an equality to another. The missing fact is a name for the element, which is a representation change. `PlanOp::describe` becomes a method so a test can state "the same pushed nodes" without naming node indices, which two orderings do not share. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fold rule declines a constant in the object position -- "a constant
object is a filter, a different rule, a different claim" -- and this is
that rule. The match becomes a `Filter` on the scan of its subject and
the join that carried it disappears, with both nodes' claims moving to
the filter. It rewrites rather than pushes: the filter is `Engine`, and
push_comparison_filter decides whether SQL can express it, which is also
where the constant-is-the-column's-term test already lives.
The precondition is that the match's one consumer is a plain `Join`
whose other side takes *every* row from an `Sql` scan of the same star.
`inner_join_groups` cannot state that any more: it treats every
non-join as a boundary, so a filter this rule already inserted between
the scan and the join looks like a different row set, and a second
constant on one star declined. `mandatorily_feeds` asks the question of
an edited plan instead -- plain joins, filters and unnests only.
That check is not tidiness. `?t a Track . OPTIONAL { ?s a Signal } .
?s :name "BX517"` parses as a plain join of the left join with the
constant match, so the structural check passes; but a solution leaving
`?s` unbound is compatible with a pattern that binds it, so the join
keeps that row and a filter on the scan's column does not. Test
included, and it fails without the check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A `Join` between two `Sql` scans on a reference slot becomes `Sql`. The edge is `JoinEdge`'s -- one star holds the foreign key, the other is scanned as a star -- and the plan already spells it that way after the fold: one scan binds the joined variable to a slot, the other has it as its `star_var`. Preconditions and what each prevents: exactly one join variable (two stars sharing two variables are joined by more than the reference); one side scans it as a star and the other binds it to a slot of its class (a variable two stars bind as slots is a value join between columns); the slot stores a reference and not an inlined structure (there is no column holding an identifier -- `Inexact::TypedNestedStructure`), tested with the same `SlotInlineMode::Reference` the star decomposition uses; the slot is single-valued (an array of identifiers would have the join compare an element, which the plan has no name for); and both sides already run in SQL. A left join is a different operator, so no rule matches it -- tested with both of its sides pushed, so it is the rule declining rather than an unreachable frontier. The test schema gains `LineGroup`, whose `groupsLines` is a multivalued reference: `documents` is multivalued too but inlined, so without a class like this the two shapes a slot can be an array of are indistinguishable in the fixture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`tier_one_rules` is the four rules in the order 28d lists them, and the cross-rule tests are about the set rather than about any one of them, over a corpus that includes both filter orderings, a chain of reference joins, an OPTIONAL, a BIND, a sub-select, VALUES and all four query forms: * **the rule order does not decide the fixpoint.** Every rule is monotone -- two remove a `match`, two turn an `Engine` node `Sql`, none does the reverse -- so listing them backwards reaches the same plan, printout for printout. The order buys rounds, not results. * **refinement never changes the obligations.** Same list, same count, same residual, each still claimed exactly once. * **every invariant after every application of every rule**, not only behind the driver's debug assertion. * **the worked example ends in the shape 28d draws**, including that the grouping is refused by the frontier invariant rather than by a verdict -- the assertion names the engine filter that is its input. Drop the regex and the frontier reaches the grouping's input. One thing the order test had to learn: two *parses* of one query are not the same plan, because spargebra names the internal variable of `(COUNT(*) AS ?n)` freshly each time. The test refines two clones of one naive plan instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The unchecked render was public, so any future caller could obtain a `SqlCondition` without the half of the test that needs a schema. That is the trap this architecture exists to avoid, so the shape test is now `Expr::sql_shape_unchecked` and `pub(crate)`, and the only public entry is `Expr::to_sql(schema, class_of_star)`, which asks both halves: does SQL have this shape, and is the constant the term the column's values render as. The term test moves out of the filter rule into `sparql_refine`, so both rules that push a constant ask it at the same place -- the constant object rule now builds its condition and calls `to_sql`, rather than carrying its own copy of the judgement. `push_form_of_path` is the same question about a value further inside a record, which the nested-path work needs and which `resolve_column` already walks. A star the class map does not name declines: an address nobody can resolve is not a condition. Two tests state the split: the shape half accepts every constant these four cases compare, and the public entry declines all four (an enum code, an enum IRI, a tagged literal, a non-canonical number). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…push `(star, slot_path)` named three different values and could not say which, so both rules declined every multivalued slot. `SlotReading` is that missing fact, on `Expr::Slot` and on `SqlCondition`: * `Column` -- the column's own text. Only right for a single-valued slot. * `AnyElement` -- some element of the array. `?s :trafficKinds "m"` binds nothing and matches a record once however many values it holds, which is a containment test and exactly what `Star::multivalued_fields` tells its renderer to do. So the constant-object rule pushes it, and the rewrite stays cardinality-preserving. * `BoundElement` -- the element a `PlanOp::Unnest` bound to a variable. One row per value, which is what `?s :trafficKinds ?k . FILTER(?k = "m")` means: one solution per matching value, not every value of a matching record. So the filter rule pushes it, and only where that unnest is below the landing site -- before the fan-out the element has no name. `PlanOp::Unnest` therefore carries `var`, the variable the folded match read the slot into, which is what a `BoundElement` refers to. That makes the fifth invariant stronger in both directions: the unnest restoring a slot's fan-out must bind *that slot's variable* (an unnest under the wrong name leaves a condition addressing an element nothing bound), and an unnest no scan folded is now a defect of its own, `StrayFanout` -- it multiplies rows by an array no row set has. Both have a test that drives a bad rule through the driver. One shape the star decomposition refuses outright is now expressible: `:trafficKinds "m" ; :trafficKinds ?k` carries both readings of one slot (`Inexact::ConstantAndVariableOnSlot`), because two nodes can say what one `Star` cannot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s it I argued this was better left derivable; the counter-argument is right. `on` plus the scans do determine the direction, but a consumer that has to re-derive it is a second derivation of one fact, which is where a renderer comes to disagree with the plan it renders -- and today's `JoinEdge` already carries `right_slot` for exactly that reason. So `PlanOp::Join` gains `reference: Option<ReferenceEdge>`, named for what each end is (`referenced` / `holder` / `slot`) because "left" and "right" also name the plan node's sides and the two need not line up. `None` in a naive plan and on any join no rule pushed: a natural join on a shared variable is not necessarily a reference. The printout shows the edge, so a reader need not reconstruct it either. And it is checkable, which was the other half of my objection. Invariant 6, `reference_joins_agree`: the referenced star is scanned on one side, the holder on the other, the recorded slot is the one that scan bound to the joined variable, and the join is on that variable alone. Whether the slot is really a foreign key stays the rule's question -- a plan has no schema -- but a reversed edge was otherwise a wrong join in a plan every invariant passed. Test drives a rule that swaps the ends. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A value inside an inlined structure is still a value the SQL side can read -- `?s :location ?loc . ?loc :longitude ?lon` reads `["location", "longitude"]`, which the star decomposition calls a `PathFilter` and renders by walking into the JSON. `ScanSlot.slot` becomes `ScanSlot.path`, so the scan's vocabulary matches `Expr::Slot.slot_path` and `PathFilter.slot_path` and the nested-fold rule has somewhere to record what it folded. Inert on its own: every path the rules build today is one element long. Two places now say so deliberately rather than by construction -- a foreign key is a column of the record, so `ReferenceEdge`'s slot and the join rule's key both require a one-element path, since a reference inside an inlined structure is not something `JoinEdge`'s single slot name can address. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The last capability gap against the planner that serves queries today: `?s :location ?loc . ?loc :longitude ?lon` reads one value two slots down, which `Star::path_filters` pushes by walking into the JSON and the refined plan rejected outright -- so the whole star declined any filter on `?lon`. `FoldNestedMatchIntoPath` folds such a read into the scan as a path, and the join that carried it collapses the way the type fold's joins do. `subject_site` is the shared answer to "where does this subject live" -- a star, or a structure inside a scanned record -- so the constant rule resolves `?loc :longitude 5` through the same resolution and pushes it as a path filter. One place decides, so the two cannot disagree. Preconditions, each preventing a wrong answer rather than being cautious: * a *reference* hop is not walked into. The foreign key holds an identifier, so the value beyond it lives in another record: that is a join, and reading it as a path reads a column that does not exist. The star decomposition refuses the same shape. * a multivalued hop is not walked into, and neither is a multivalued nested slot. The first would address a field of one array element -- a reading nothing renders; the second would owe an unnest, and a scan that owes one can only be built by `scan_with_fanout`, which this rule does not go through. Lifting it means rebuilding the scan, not editing it. * the match's one consumer is a plain join on the subject, whose other side takes every row from that scan -- a nested read inside an `OPTIONAL` is not a column of the preserved side. Paths compose: the rule applies to its own output, so three hops fold in three applications. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…today
The gate on stage 3: for every corpus query, whatever today's SQL pass
claims the refined plan's `Sql` nodes claim too, and whatever conditions
today's star decomposition pushes the refined plan pushes too. Stated as
containment in that direction, because the refined plan legitimately
claims more; switching the endpoint onto a planner that pushed less
would regress answers or lose pushdown.
Comparing it found one real gap and closed it. A `VALUES` over a
variable a scan already binds is a membership test -- today's star makes
an `In` condition of it -- and the refined plan left the block and its
join with the engine. `ValuesBecomesFilter` is that rule, and its
precondition is the discriminator `Obligation::Values`' own doc implies:
a `VALUES` binding a variable nothing else binds adds rows and columns
and is *not* a filter, so only one whose variable a scan below binds
becomes one. It also refuses a bag (`{ "a" "a" }` returns each solution
twice where `IN` returns it once), an `UNDEF`, and more than one
variable.
Both dimensions of the test were verified to fail when broken: emptying
the gap table fails on the OPTIONAL query, and dropping the nested-path
rule fails on the path-filter condition.
Two gaps stated rather than smoothed:
* `KNOWN_GAPS` has one entry, and it is not a pushdown difference but a
disagreement about what claiming a triple *means*. A read inside an
`OPTIONAL` reaches today's star as an `optional_fields` entry: the
prefetch delivers the column with no existence check, and the pass
claims the triple because the data reaches oxigraph. The refined plan
claims an obligation when a node takes care of it, and a fetch that
requires nothing enforces no triple. I would argue against closing it
by claiming it -- `plan_query`'s own comment refuses to let a pass say
it enforced something it did not. Stage 3 picks a meaning; today's
needs a scan slot that is fetched without being required, which is a
representation change.
* a constraint on a *star* variable (`VALUES ?s { ... }`,
`FILTER(?s = <iri>)`) is a constraint on the record's identity, and no
`Expr::Slot` addresses that -- today's planner pushes it as
`identifier_values`. Written on the identifier slot itself it does
push. Noted where `Visible` marks star variables ambiguous.
Tier-two obligations (group, aggregate, order, slice, distinct) are
excluded by kind rather than compared: they collapse or reorder rows, no
rule here moves one, and comparing them would only assert that tier two
is unbuilt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…arity Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
32 commits. A planner that turns a SPARQL query into an execution plan, the soundness work that makes trusting such a plan defensible, and enough of an operator algebra that optimisation can later be a plan → plan transform.
Consumed by consolidator-server MR !861, which renders the plan to SQL and compares both routes with a differential oracle over real rows.
Why
An aggregate has no selective filter — it touches every object of its class by definition — so the existing route (fetch objects, let oxigraph aggregate) cannot serve one at any realistic size. This answers grouping and aggregation in SQL where that can be proven equivalent, and refuses in a way its author can act on where it cannot.
The artifact
plan_query(query, schema_view)→ExecutionPlan: one parse, one scope, one analysis, one artifact. Previouslysparql_scopeandsparql_pushdowneach parsed the query and each answered half the question, so the two halves could describe different queries with nothing to notice.The plan is a ledger. Every query decomposes into obligations — each triple, filter, group, aggregate, ordering, slice — and each must be discharged by a pass or listed as residual.
is_accounted()false means the plan answers a different question than the one asked, and the caller must refuse to run it.ledger_balances()asserts each obligation appears exactly once.An
SqlPassis a flat list of operators:scan,filter,join,unnest,group,sort,distinct,slice,project, each claiming the obligations it discharges, inputs as indices that always precede their node. It is the pass's only description — it carried a star decomposition and a solution spec until every consumer read the nodes, and three descriptions of one pass is how a reader comes to use the stale one.Three facts live on the nodes because a renderer cannot recover them, and getting any wrong is a wrong answer rather than an error:
numericon a filter — unreadable from the path (the same slot name on two classes may differ; a value inside a structure is not in the record's own slot list). Comparing a number as text makes'9' >= '10'true.is_optionalon a scan — decides the join type above it and whether the scan's conditions must tolerate an unmatched row. Rendered as required, anOPTIONALblock drops rows the query keeps.Enforcement::{Enforces, Narrows}— when an engine pass decides the answer, SQL may narrow rows while the engine stays authoritative for term semantics; when SQL answers alone the same filter is the answer. Previously inferable only by comparing which pass claimed the obligation.Reading a plan
Display, and__str__from Python. Real output.Answered entirely in SQL — one pass, every obligation discharged, no engine.
unnestis there because that slot is multivalued: rows must fan out or the count is per record instead of per value.SQL narrows, the engine finishes — a
REGEXSQL cannot express. The scan claims the triples, the engine claims the filter and says why it is needed.A narrowing filter and a fetch bound — the constant is applied in SQL but marked
narrows, since the engine reapplies it with term semantics. Thelimit 20node is the fetch bound, claiming nothing: the engine still applies the query's ownLIMIT, which is whyo3stays with it while the scan claimso2.A refused aggregate — still a usable plan, since the engine answers it, but the refusal travels on the artifact so one call gives the caller both the route and what to tell whoever wrote the query.
Soundness
Most of the
fix(...)commits are one theme: a plan must not claim more than it enforces. A pass that says it applied a filter it silently dropped produces a plausible wrong answer, which is worse than an error. So every drop site records its cause, a triple stays inexact until something claims it, and the cause list is generated from the planner rather than written down — twice a docstring had drifted, advertising causes that could not occur and omitting one that could.Notable in that vein:
GROUP BY,ORDER BYandDISTINCT(the first commit, reviewable alone).… GROUP BY ?v LIMIT 10counted ten objects instead of returning ten groups;SELECT DISTINCT … LIMIT 10deduplicated an arbitrary ten. No error either way. Live bug onmain.HAVINGis reported as a refusal rather than quietly unrecognised.Terms
sparql_termsresolves, per column, how stored text becomes an RDF term — plain, typed, language-tagged, IRI, or an enum'smeaning— since the SQL route must answer in the same terms oxigraph would or the same question answers differently per route.It reads the already-public
RangeInfo, so numeric-ness comes from the resolved datatype IRI: a schema-defined type withtypeof: integeris a number, where matching the range's spelling refused it. Only the precedence (lang beats datatype, enum meaning beats literal) is restated fromlinkml_runtime::turtle's private helpers, and MR !861's oracle pins it end to end — no upstream change turned out to be needed.Refusals are part of the contract
Every refusal carries a closed-set
code, adetailin data-model terms, optionallyat(the variable or pattern), andinstead— a supported shape. The intent is that the subset's boundary is learned from the errors rather than from a document.Optimisation
Not built, deliberately: the planner emits one shape per query, so there is nothing to choose between, and the rewrites worth having first were all "SQL should handle this filter" — achieved by teaching the planner the filter. The structure is what a rule would need: move a node, move its claims, re-check
ledger_balances()andis_well_formed(). Missing for a cost-based rule: per-node cardinality and statistics.Scope
Single-class and reference-joined questions; grouping on scalar slots and on values inside nested structures;
COUNT/SUM/AVG/MIN/MAX; comparison and enum filters;HAVING/ORDER BY/LIMIT/OFFSET/DISTINCT.GROUP_CONCATandSAMPLEare refused permanently — SPARQL leaves their order unspecified, so the two routes could not be held to the same answer.Tests
235 pass;
cargo fmt --all -- --check,cargo clippy --all-targets --all-features -- -D warningsandcargo testall clean. The negative corpus asserts each refusal's code and that it carries a detail and a hint, so a message degrading to "unsupported" fails a test. On the consumer side, 702 pass with the differential oracle comparing both routes' answers over real rows.🤖 Generated with Claude Code