Skip to content

fix(shard-engine): quote json path segments that misparse - #371

Merged
prisis merged 1 commit into
alphafrom
fix/json-path-field-quoting
Aug 8, 2026
Merged

prisis merged 1 commit into
alphafrom
fix/json-path-field-quoting

Conversation

@prisis

@prisis prisis commented Aug 8, 2026 •

Copy link
Copy Markdown
Member

The defect

The DO row store splices a field name into a json_extract path in two places, and both got the escaping wrong — in opposite directions.

documentPath (packages/shard-engine/src/do-sql.ts) escaped the name for the SQL string literal and not for SQLite's JSON-path grammar:

return `json_extract(${prefix}${DOC_COLUMN}, '$.${field.replaceAll("'", "''")}')`;

SQLite reads a bare .label up to the next . or [, so a field named a.b becomes $.a.b and resolves as the nested key b under a, not the top-level key a.b. Nothing raises. Verified against a real build (SQLite 3.53.0) on a document holding both:

path reads
$.a.b "nested" — the wrong slot
$."a.b" "flat" — correct

Same class for [. This feeds jsonPath / jsonPathSql / qualifiedJsonPath, so it reached where compilation, ORDER BY, aggregates, the EXISTS correlation refs, and the CREATE INDEX expressions.

resolveColumnExpression (packages/shard-engine/src/introspect.ts) did quote, but escaped " by doubling it — column.replaceAll('"', '""') — as if it were a SQL identifier. Inside a quoted JSON-path label SQLite applies JSON string escapes, so "" is not an escape at all and the path resolves to NULL. \ has the same problem, and neither site escaped it:

path reads a doc with key q"x / back\slash
$."q""x" null
$."q\"x" the value
$."back\slash" null
$."back\\slash" the value

Reachability — what I found

This is the part worth reading, and it went the less comfortable way.

No layer between the wire and jsonPathSql compares a field name against definition.shape. The build-time guard (FIELD_NAME_RE in packages/codegen/src/parse-validator.ts:247) does reject a non-identifier schema field name, but the shard engine receives the schema object at runtime and re-derives paths from it; nothing at the runtime boundary re-checks, and nothing checks names that never came from the schema at all.

Every framework-owned inbound path is nevertheless closed:

path field-name origin reaches the path builder?
defineListArgs (packages/server/src/list-args.ts) allow-list, compile-checked yes, declared names only — sanitizeWhere iterates from the allow-list, and toQueryArgs re-filters orderBy against sortable because it is exported
Admin RPCs (readTablePage, facetColumn, deleteRows) arbitrary, admin-bearer gated no — they route to introspect.ts's own resolver, which binds the path as ?
runSql arbitrary SQL, admin-bearer gated no
@lunora/db no field names on the wire at all no
@lunora/studio grid column id no — lands on readTablePage
Relation / EXISTS correlation refs relationMap, from the schema yes, schema-only
Secondary indexes, TTL, soft-delete schema yes, schema-only

The one open door is the plain ctx.db.<table>.findMany({ where, orderBy }) API. where-sql.ts documents itself as a storage-blind compiler and is exactly that — compileNode iterates Object.entries(where) and hands every non-structural key to strategy.fieldRef, which is jsonPathSql. orderBy is the same through query-args.ts. So a procedure that spreads one of its own arguments into where or orderBy reaches the path builder with a string straight off the wire, gated only by whatever v.* validator the author declared for it. Nothing in packages/, examples/, or templates/ does this today — it is a latent capability of the API, not a live bug in-repo. RLS does not help: assertOrderByAllowed in the mask middleware rejects masked columns; an unknown name passes.

Severity, plainly. It is not an injection, and never was — the ' doubling closed the SQL-string escape and drizzle binds every value. But before this change such a name steered json_extract into the document's nested keys, which is a filter oracle over data the caller was never handed: where: { "settings.apiKey": { gt: "m" } } on a doc whose settings object is not in the projected result set. Correct quoting closes that outright — $."settings.apiKey" names a top-level key that does not exist, so the predicate is simply unsatisfied rather than traversing.

So: a correctness bug for schema-declared names, and a narrow information-disclosure vector on the one API surface that forwards caller-supplied names. Quoting at the choke point is the fix for both, which is why there is no separate validation layer here — a rejecting guard would additionally contradict the fact that quoting makes such a field work.

The fix

shared/json-path-segment.ts — one definition of the rule, for the same reason and under the same no-dependency-edge constraint as the neighbouring shared/quote-identifier.ts. Two copies free to drift is precisely how these two sites ended up wrong in different ways.

const BARE_JSON_PATH_LABEL = /^[A-Za-z_$][\w$]*$/u;

export const jsonPathSegment = (field: string): string =>
    BARE_JSON_PATH_LABEL.test(field) ? field : `"${field.replaceAll("\\", String.raw`\\`).replaceAll('"', String.raw`\"`)}"`;

Why it quotes conditionally

These paths are the expression text of the store's CREATE INDEX statements, and SQLite matches an expression index to a query by comparing that text. Quoting unconditionally would leave every index already on disk unmatched and silently unused — every affected query degrades to a scan with nothing failing.

The bare shape above is deliberately the same identifier shape codegen already enforces on every schema-declared field, so every path this store has ever emitted keeps its exact spelling. That is asserted, not assumed: do-sql.json-path.test.ts pins the literal strings, and reverting only do-sql.ts leaves those four assertions green while the five defect assertions fail.

introspect.ts's path is a bound parameter, so it was never index-matching and the conditional quoting is free there.

Tests

packages/shard-engine/__tests__/do-sql.json-path.test.ts (new, 9 tests):

  • identifier-safe fields emit the historical bare path byte for byte, and _id / id / _creationTime keep their stored-column mapping;
  • ., [, space, ', ", \ each produce the quoted form, with both JSON escapes;
  • against a real node:sqlite build, jsonPath("a.b") reads the flat key while jsonPath("a") still reads the nested object, and every awkward name round-trips.

introspect.test.ts gains a readTablePage case ordering by a.b and by q"x.

Negative control run both ways: reverting src/do-sql.ts fails 5 of 9 and passes the 4 byte-identity ones; reverting src/introspect.ts fails the new q"x ordering.

Gates

@lunora/shard-engine 992 ✓ · @lunora/do 522 ✓ · @lunora/codegen 1068 ✓ · @lunora/server 570 ✓ · lint:types ✓ · lint:eslint --max-warnings=0 ✓ · api:check (47 snapshots, unchanged — the helper is not public surface) ✓ · lint:package-json ✓ · build:packages ✓.

dist:check fails on packages/studio's dev-build JSX factory, which is what build:packages (not build:packages:prod) produces and is unrelated to this change.

Note for fix/265-wire-codec-query-parity

That branch's reprojection-backfill.ts carries a local jsonPathSegment with a comment calling this a known gap. It should drop the local copy for the shared one on merge — and note the local copy does not escape \, which is required.

Summary by CodeRabbit

  • Bug Fixes

    • Fixed queries involving document fields with dots or other special characters.
    • Improved handling of untrusted field names in raw SQL expressions.
    • Ensured document field paths are escaped correctly for reliable SQLite JSON access.
  • Documentation

    • Clarified escaping requirements when embedding raw SQL with document field names.

The DO row store splices a field name into a `json_extract` path in two
places, and both got the escaping wrong, in opposite directions.

`documentPath` in `do-sql.ts` escaped the name for the SQL string literal
but not for SQLite's JSON-path grammar, so a field carrying a `.` or `[`
addressed a different value than the one it named: `a.b` became `$.a.b`,
which resolves as the nested key `b` under `a` rather than the top-level
key `a.b`. Nothing raised — the query read the wrong slot, or none. This
feeds `jsonPath` / `jsonPathSql` / `qualifiedJsonPath`, so it reached
`where` compilation, `ORDER BY`, aggregates, the EXISTS correlation refs,
and the `CREATE INDEX` expressions.

`resolveColumnExpression` in `introspect.ts` did quote, but escaped `"` by
doubling it as if it were a SQL identifier. `""` is not a JSON-path escape
— SQLite applies JSON string escapes inside a quoted label, so the path
resolved to NULL. An unescaped `\` does the same, and neither site handled
it.

Hoist the rule to `shared/json-path-segment.ts` so there is one definition
rather than two copies free to drift again — the same reasoning, and the
same no-dependency-edge constraint, as `shared/quote-identifier.ts`. It
escapes `\` and `"` and quotes only when the bare form would misparse,
keyed on the identifier shape `@lunora/codegen`'s `parseValidatorObject`
already enforces on every schema field. Every path the store has emitted
keeps its exact spelling, so the expression indexes already on disk keep
matching their queries.

Reachability, since it decides the severity: no layer between the wire and
`jsonPathSql` compares a field name against `definition.shape`.
`defineListArgs` rebuilds `where` from its allow-list and re-checks
`orderBy`, the admin RPCs route to `introspect.ts`'s own resolver behind
the admin bearer, `@lunora/db` sends no field names, and relations/indexes
/TTL are schema-sourced. But `ctx.db.<table>.findMany({ where, orderBy })`
passes its keys straight through `where-sql.ts`'s storage-blind compiler,
so a procedure that spreads an argument into `where` reaches the path
builder with a name off the wire. Not an injection — `'` doubling and
drizzle binding already closed that — but before this change such a name
steered `json_extract` into the document's nested keys, which is a filter
oracle over data the caller was never handed. Correct quoting closes it:
`$."a.b"` names a key that does not exist rather than traversing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
@netlify

netlify Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit 0d810a8
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a7760b382011c00086bc93f
😎 Deploy Preview https://deploy-preview-371--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@coderabbitai

coderabbitai Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 63a34343-9bc9-466b-adb9-5a929fb7e8bb

📥 Commits

Reviewing files that changed from the base of the PR and between 4361cbc and 0d810a8.

⛔ Files ignored due to path filters (3)
  • packages/shard-engine/__tests__/do-sql.json-path.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/introspect.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • shared/json-path-segment.ts is excluded by none and included by none
📒 Files selected for processing (2)
  • packages/shard-engine/src/do-sql.ts
  • packages/shard-engine/src/introspect.ts

Walkthrough

The change applies shared JSON-path escaping to document field names in SQL construction and column introspection. Documentation now describes dual escaping for unvalidated field names.

Changes

JSON path escaping

Layer / File(s) Summary
Safe document path construction
packages/shard-engine/src/do-sql.ts, packages/shard-engine/src/introspect.ts
documentPath and resolveColumnExpression use jsonPathSegment before SQL literal escaping. The jsonPathSql documentation describes both escaping layers and unvalidated field names.

Estimated code review effort: 1 (Trivial) | ~5 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: fixing JSON path segment quoting in the shard engine.
Description check ✅ Passed The description thoroughly covers the defect, fix, scope, tests, validation gates, and an unrelated failure; some template sections are not explicit.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/json-path-field-quoting

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@prisis

prisis commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

Merge-order note, mirrored from #365: that branch carries a local jsonPathSegment in reprojection-backfill.ts:56 which this PR supersedes — it does not escape \\ and quotes unconditionally. Neither is reachable there (the backfill's field names are schema keys, which parse-validator.ts:247 already constrains to identifiers), but whichever of #365 / #371 merges second should delete the local copy and import shared/json-path-segment.

@prisis
prisis merged commit 36f3fe4 into alpha Aug 8, 2026
44 checks passed
@prisis
prisis deleted the fix/json-path-field-quoting branch August 8, 2026 17:13
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (alpha@4361cbc). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff            @@
##             alpha     #371   +/-   ##
========================================
  Coverage         ?   87.07%           
========================================
  Files            ?     1170           
  Lines            ?    63254           
  Branches         ?    15555           
========================================
  Hits             ?    55079           
  Misses           ?     7646           
  Partials         ?      529           
Files with missing lines Coverage Δ
packages/shard-engine/src/do-sql.ts 95.08% <100.00%> (ø)
packages/shard-engine/src/introspect.ts 95.93% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed

codspeed Bot commented Aug 8, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 2.2%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 2 regressed benchmarks
✅ 249 untouched benchmarks
⏩ 10 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ pure diff (unchanged) — 10 rows 245.8 µs 308 µs -20.19%
❌ findMany — plain table (no scope) 77.8 ms 88.2 ms -11.73%
⚡ 1 after-insert no-op trigger 483.5 µs 418.6 µs +15.5%
⚡ flat 3 primitives (the notify.send attribute shape) 62.5 µs 55.5 µs +12.44%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing fix/json-path-field-quoting (0d810a8) with alpha (4361cbc)

Open in CodSpeed

Footnotes

  1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — 0d810a8d Deployed Aug 8, 2026 by prisis via Benchmarks #1556
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants