Repository navigation
fix(shard-engine): quote json path segments that misparse - #371
Conversation
The DO row store splices a field name into a `json_extract` path in two
places, and both got the escaping wrong, in opposite directions.
`documentPath` in `do-sql.ts` escaped the name for the SQL string literal
but not for SQLite's JSON-path grammar, so a field carrying a `.` or `[`
addressed a different value than the one it named: `a.b` became `$.a.b`,
which resolves as the nested key `b` under `a` rather than the top-level
key `a.b`. Nothing raised — the query read the wrong slot, or none. This
feeds `jsonPath` / `jsonPathSql` / `qualifiedJsonPath`, so it reached
`where` compilation, `ORDER BY`, aggregates, the EXISTS correlation refs,
and the `CREATE INDEX` expressions.
`resolveColumnExpression` in `introspect.ts` did quote, but escaped `"` by
doubling it as if it were a SQL identifier. `""` is not a JSON-path escape
— SQLite applies JSON string escapes inside a quoted label, so the path
resolved to NULL. An unescaped `\` does the same, and neither site handled
it.
Hoist the rule to `shared/json-path-segment.ts` so there is one definition
rather than two copies free to drift again — the same reasoning, and the
same no-dependency-edge constraint, as `shared/quote-identifier.ts`. It
escapes `\` and `"` and quotes only when the bare form would misparse,
keyed on the identifier shape `@lunora/codegen`'s `parseValidatorObject`
already enforces on every schema field. Every path the store has emitted
keeps its exact spelling, so the expression indexes already on disk keep
matching their queries.
Reachability, since it decides the severity: no layer between the wire and
`jsonPathSql` compares a field name against `definition.shape`.
`defineListArgs` rebuilds `where` from its allow-list and re-checks
`orderBy`, the admin RPCs route to `introspect.ts`'s own resolver behind
the admin bearer, `@lunora/db` sends no field names, and relations/indexes
/TTL are schema-sourced. But `ctx.db.<table>.findMany({ where, orderBy })`
passes its keys straight through `where-sql.ts`'s storage-blind compiler,
so a procedure that spreads an argument into `where` reaches the path
builder with a name off the wire. Not an injection — `'` doubling and
drizzle binding already closed that — but before this change such a name
steered `json_extract` into the document's nested keys, which is a filter
oracle over data the caller was never handed. Correct quoting closes it:
`$."a.b"` names a key that does not exist rather than traversing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (3)
📒 Files selected for processing (2)
WalkthroughThe change applies shared JSON-path escaping to document field names in SQL construction and column introspection. Documentation now describes dual escaping for unvalidated field names. ChangesJSON path escaping
Estimated code review effort: 1 (Trivial) | ~5 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for following the naming conventions! 🙏 |
|
Merge-order note, mirrored from #365: that branch carries a local |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## alpha #371 +/- ##
========================================
Coverage ? 87.07%
========================================
Files ? 1170
Lines ? 63254
Branches ? 15555
========================================
Hits ? 55079
Misses ? 7646
Partials ? 529
🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 2.2%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | pure diff (unchanged) — 10 rows |
245.8 µs | 308 µs | -20.19% |
| ❌ | findMany — plain table (no scope) |
77.8 ms | 88.2 ms | -11.73% |
| ⚡ | 1 after-insert no-op trigger |
483.5 µs | 418.6 µs | +15.5% |
| ⚡ | flat 3 primitives (the notify.send attribute shape) |
62.5 µs | 55.5 µs | +12.44% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing fix/json-path-field-quoting (0d810a8) with alpha (4361cbc)
Footnotes
-
10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
The defect
The DO row store splices a field name into a
json_extractpath in two places, and both got the escaping wrong — in opposite directions.documentPath(packages/shard-engine/src/do-sql.ts) escaped the name for the SQL string literal and not for SQLite's JSON-path grammar:SQLite reads a bare
.labelup to the next.or[, so a field nameda.bbecomes$.a.band resolves as the nested keybundera, not the top-level keya.b. Nothing raises. Verified against a real build (SQLite 3.53.0) on a document holding both:$.a.b"nested"— the wrong slot$."a.b""flat"— correctSame class for
[. This feedsjsonPath/jsonPathSql/qualifiedJsonPath, so it reachedwherecompilation,ORDER BY, aggregates, the EXISTS correlation refs, and theCREATE INDEXexpressions.resolveColumnExpression(packages/shard-engine/src/introspect.ts) did quote, but escaped"by doubling it —column.replaceAll('"', '""')— as if it were a SQL identifier. Inside a quoted JSON-path label SQLite applies JSON string escapes, so""is not an escape at all and the path resolves to NULL.\has the same problem, and neither site escaped it:q"x/back\slash$."q""x"null$."q\"x"$."back\slash"null$."back\\slash"Reachability — what I found
This is the part worth reading, and it went the less comfortable way.
No layer between the wire and
jsonPathSqlcompares a field name againstdefinition.shape. The build-time guard (FIELD_NAME_REinpackages/codegen/src/parse-validator.ts:247) does reject a non-identifier schema field name, but the shard engine receives the schema object at runtime and re-derives paths from it; nothing at the runtime boundary re-checks, and nothing checks names that never came from the schema at all.Every framework-owned inbound path is nevertheless closed:
defineListArgs(packages/server/src/list-args.ts)sanitizeWhereiterates from the allow-list, andtoQueryArgsre-filtersorderByagainstsortablebecause it is exportedreadTablePage,facetColumn,deleteRows)introspect.ts's own resolver, which binds the path as?runSql@lunora/db@lunora/studioreadTablePagerelationMap, from the schemaThe one open door is the plain
ctx.db.<table>.findMany({ where, orderBy })API.where-sql.tsdocuments itself as a storage-blind compiler and is exactly that —compileNodeiteratesObject.entries(where)and hands every non-structural key tostrategy.fieldRef, which isjsonPathSql.orderByis the same throughquery-args.ts. So a procedure that spreads one of its own arguments intowhereororderByreaches the path builder with a string straight off the wire, gated only by whateverv.*validator the author declared for it. Nothing inpackages/,examples/, ortemplates/does this today — it is a latent capability of the API, not a live bug in-repo. RLS does not help:assertOrderByAllowedin the mask middleware rejects masked columns; an unknown name passes.Severity, plainly. It is not an injection, and never was — the
'doubling closed the SQL-string escape and drizzle binds every value. But before this change such a name steeredjson_extractinto the document's nested keys, which is a filter oracle over data the caller was never handed:where: { "settings.apiKey": { gt: "m" } }on a doc whosesettingsobject is not in the projected result set. Correct quoting closes that outright —$."settings.apiKey"names a top-level key that does not exist, so the predicate is simply unsatisfied rather than traversing.So: a correctness bug for schema-declared names, and a narrow information-disclosure vector on the one API surface that forwards caller-supplied names. Quoting at the choke point is the fix for both, which is why there is no separate validation layer here — a rejecting guard would additionally contradict the fact that quoting makes such a field work.
The fix
shared/json-path-segment.ts— one definition of the rule, for the same reason and under the same no-dependency-edge constraint as the neighbouringshared/quote-identifier.ts. Two copies free to drift is precisely how these two sites ended up wrong in different ways.Why it quotes conditionally
These paths are the expression text of the store's
CREATE INDEXstatements, and SQLite matches an expression index to a query by comparing that text. Quoting unconditionally would leave every index already on disk unmatched and silently unused — every affected query degrades to a scan with nothing failing.The bare shape above is deliberately the same identifier shape codegen already enforces on every schema-declared field, so every path this store has ever emitted keeps its exact spelling. That is asserted, not assumed:
do-sql.json-path.test.tspins the literal strings, and reverting onlydo-sql.tsleaves those four assertions green while the five defect assertions fail.introspect.ts's path is a bound parameter, so it was never index-matching and the conditional quoting is free there.Tests
packages/shard-engine/__tests__/do-sql.json-path.test.ts(new, 9 tests):_id/id/_creationTimekeep their stored-column mapping;.,[, space,',",\each produce the quoted form, with both JSON escapes;node:sqlitebuild,jsonPath("a.b")reads the flat key whilejsonPath("a")still reads the nested object, and every awkward name round-trips.introspect.test.tsgains areadTablePagecase ordering bya.band byq"x.Negative control run both ways: reverting
src/do-sql.tsfails 5 of 9 and passes the 4 byte-identity ones; revertingsrc/introspect.tsfails the newq"xordering.Gates
@lunora/shard-engine992 ✓ ·@lunora/do522 ✓ ·@lunora/codegen1068 ✓ ·@lunora/server570 ✓ ·lint:types✓ ·lint:eslint --max-warnings=0✓ ·api:check(47 snapshots, unchanged — the helper is not public surface) ✓ ·lint:package-json✓ ·build:packages✓.dist:checkfails onpackages/studio's dev-build JSX factory, which is whatbuild:packages(notbuild:packages:prod) produces and is unrelated to this change.Note for
fix/265-wire-codec-query-parityThat branch's
reprojection-backfill.tscarries a localjsonPathSegmentwith a comment calling this a known gap. It should drop the local copy for the shared one on merge — and note the local copy does not escape\, which is required.Summary by CodeRabbit
Bug Fixes
Documentation