Skip to content

fix(studio,shard-engine): decode wire-tagged doc columns on the display read paths - #339

Merged
prisis merged 6 commits into
alphafrom
advisor/300-decode-doc-read-paths
Aug 4, 2026
Merged

prisis merged 6 commits into
alphafrom
advisor/300-decode-doc-read-paths

Conversation

@prisis

@prisis prisis commented Aug 3, 2026 •

Copy link
Copy Markdown
Member

Follow-on to #338. Rewritten after review — the plan this executes was wrong about its own primary surface, and my first cut of the fix was a regression. History kept so the correction is auditable.

Plan: plans/300-decode-doc-read-paths.md.

What the plan claimed, and what is actually true

The plan said three display paths parse __doc__ with bare JSON.parse, so a v.bigint() column renders as its raw tag (["$lunora.wire$","bigint","1000"]) where an amount belongs.

For the data browser grid that was false. readTablePage's result goes back through jsonResponse, and LunoraClient decodes the entire response (decodeWire(body.result)), so the tagged array is already a real bigint before formatCell runs. The grid has always rendered 1000.

And decoding there actively breaks it. jsonResponse is the one DO result path that does not encodeWire (contrast the user dispatch and the WS push, which both do). Verified by execution:

Response.json({result:{rows:[{amountMinor:1000n}]}})     -> TypeError: Do not know how to serialize a BigInt
Response.json({result:{rows:[{blob:new ArrayBuffer(3)}]}}) -> {"result":{"rows":[{"blob":{}}]}}

The throw becomes a redacted 500, so every v.bigint() table would have become unbrowsable, and bytes would have been silently emptied. That decode is reverted.

A test now pins the invariant rather than trusting a comment: the page must stay JSON-serializable, and the tagged form must survive to decode as 1000n across a simulated wire. It fails with the decode reinstated (TypeError: Do not know how to serialize a BigInt).

What is genuinely fixed

Path Why it is real
use-data-browser.tsx rowDocument Parses a raw __doc__ string. The client's decode leaves strings alone, so this one really does see the tag.
payments-panel.tsx readField Same shape. Inert today (the panel never reads paymentSessions), kept for codec agreement — now a one-line call, so it costs nothing.
The JSON editor A pre-existing defect: JSON.stringify throws on a bigint, so editing any row of a bigint table already failed. Prefill now encodes and save decodes, making an untouched save byte-identical to what was stored.
formatCell bytes A decoded ArrayBuffer genuinely reaches it, and rendered as a bare {}.

Also from review

  • The "is this a document" predicate existed in five near-copies and had already drifted (one rejected null-prototype objects; the others accepted them). Now isPlainObject + decodeDocument in shared/wire-codec.ts, next to the encoder that defines the term. Net deletion.
  • formatCell now reuses the formatBytes that already lived 70 lines below it — a 2 MB blob reads <bytes: 2.0 MB>, not <bytes: 2097152>.
  • The editor reports a JSON syntax error and a malformed tag separately instead of labelling a codec RangeError "Invalid JSON".
  • Root-guard-by-prototype (CodeRabbit's catch, a real regression in my first cut): a __doc__ whose root is a tagged value parses as an array but decodes to a Uint8Array/Date/Map, which !Array.isArray waves through. A bytes root spread into {"0":1,"1":2,"2":3}.
  • Three comments corrected — a typed array stringifies to its indices, not {}.

Second commit: a pre-existing crash cluster

The review surfaced three more surfaces that serialize a row without accounting for the client-side decode. All reachable on alpha today, since paymentSessions has three v.bigint() columns:

  • JSON view (data-browser-page.tsx) — throws during render, so the whole Data page dies, not just the toggle;
  • Export page → JSON (grid-features.tsx) — throws in the click handler; the download silently never happens;
  • hash mask strategy (mask-preview.ts) — throws and fails closed to the sentinel, so a masked bigint shows the sentinel instead of a hash.

The CSV exporter already special-cased bigint, which is how the others went unnoticed. One shared jsonRowReplacer covers the two stringify sites; mask-preview special-cases bigint only, so every other value's hash is unchanged.

Corrections to the plan's own answers

  • §9 Q2 was answered wrong. The SQL console does not share expandDocumentRows — runReadonlySql is a separate path — so it still shows raw __doc__ text. Pre-existing, out of scope, worth its own plan.
  • §1's premise for S3 was wrong: every v.bigint() in the payment schema is on paymentSessions, which the payments panel never reads.

Known, not fixed

A legacy row carrying a literal __proto__ field loses it on an untouched save — encodeWire assigns by key, so the prototype setter fires instead of the property landing. Only reachable for pre-codec rows (encodeDocJson drops it on write too).

Verification

Gates: shard-engine 981, studio 1012, client 640, do 519, lint:types clean, prettier + eslint clean. Every new behavioural test demonstrated to fail against the code it guards.

🤖 Generated with Claude Code

https://claude.ai/code/session_019WfuWNN3cWJE6beZjng347

…ay read paths

The `__doc__` blob is written through the wire codec, so a `v.bigint()` or
`v.bytes()` column is stored as a tagged array (`["$lunora.wire$","bigint",…]`).
Three display paths parsed that blob with bare `JSON.parse`, so the tag itself
reached the screen where the value belongs — on `paymentSessions.amountMinor`,
that is an amount.

- `introspect.ts`: `safeParseObject` routes through `decodeDocJson`, covering
  both its call sites. Its non-throwing contract is preserved deliberately — a
  malformed tag degrades to an unexpanded page rather than failing the read.
- `use-data-browser.tsx`: decode in `rowDocument`. Decoding alone would have
  broken the JSON editor, which is a pre-existing defect this makes visible:
  the rows now carry real bigints and `JSON.stringify` throws on one, so
  editing any row of a bigint-bearing table already failed. Fixed
  symmetrically — the editor is seeded from `encodeWire(doc)` and `writeRow`
  decodes on the way out, so an untouched save re-encodes byte-identically to
  what the shard stored.
- `payments-panel.tsx`: decode in `readField` for codec agreement. This one is
  inert today and the plan's premise for it was wrong: the panel reads only
  `subscriptions`/`events`, and every `v.bigint()` in the payment schema is on
  `paymentSessions`, which it never opens.
- `lib/internal.ts`: a decoded `ArrayBuffer` formatted to a bare `{}`, strictly
  less informative than the tag it replaced. Render `<bytes: N>` instead.

Studio reaches the codec through the bundler-inlined `shared/wire-codec` (it
already inlines `shared/log-fields`), so no package dependency was added and no
second decoder exists.

Every new behavioural test was demonstrated to fail against the base revision:
the tag surfacing instead of `1000n`, `{}` instead of `<bytes: 3>`, the payments
value vanishing to `""`, and the editor's `TypeError: Do not know how to
serialize a BigInt`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019WfuWNN3cWJE6beZjng347
@netlify

netlify Bot commented Aug 3, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit 3c9975c
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a711614794b6c0008714e7c
😎 Deploy Preview https://deploy-preview-339--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@coderabbitai

coderabbitai Bot commented Aug 3, 2026 •

Copy link
Copy Markdown

Review Change Stack

Walkthrough

The change adds shared wire-format decoding and encoding to shard introspection and Studio document flows. Bigint and binary values now decode for parsing and display, remain encoded during row editing, and decode before write operations.

Changes

Wire-format handling

Layer / File(s) Summary
Safe document parsing
packages/shard-engine/src/introspect.ts
safeParseObject decodes tagged wire values and returns undefined for malformed, undecodable, or non-object documents.
Data browser document round-trip
packages/studio/src/features/data/hooks/use-data-browser.tsx
The data browser decodes stored documents, decodes editor JSON before insert and patch operations, and encodes documents before row editing.
Studio value rendering
packages/studio/src/features/payments/payments-panel.tsx, packages/studio/src/lib/internal.ts
Payment field access decodes stored document values. Cell formatting displays binary values as byte counts.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

  • anolilab/lunora#200: Both modify payments-panel.tsx; this PR changes document decoding, while PR #200 changes payment table gating.
  • anolilab/lunora#338: Both update shard-engine and Studio document handling for shared wire-codec support.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely identifies the primary change: decoding wire-tagged document columns in Studio and shard-engine display paths.
Description check ✅ Passed The description is detailed, on-topic, and includes implementation rationale, scope, regressions avoided, known limitations, and verification results.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch advisor/300-decode-doc-read-paths

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

React Doctor found no new issues. 🎉

Reviewed by React Doctor for commit 3c9975c.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/shard-engine/src/introspect.ts`:
- Around line 861-865: Use one strict plain-record guard for decoded documents:
update safeParseObject in packages/shard-engine/src/introspect.ts (lines
861-865) to reject all non-plain roots, including byte-like objects; apply the
same guard in packages/studio/src/features/data/hooks/use-data-browser.tsx
(lines 145-150) while preserving the column-strip fallback; and validate the
decoded editor value in
packages/studio/src/features/data/hooks/use-data-browser.tsx (lines 853-859)
before assigning parsedDocument or sending the write.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 97fcfbe9-10fe-4875-9e80-c75c27e01b05

📥 Commits

Reviewing files that changed from the base of the PR and between ab0afaf and 2fabb4e.

⛔ Files ignored due to path filters (6)
  • packages/shard-engine/__tests__/introspect.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/studio/__tests__/features/data/data-browser.test.tsx is excluded by !**/__tests__/** and included by packages/**
  • packages/studio/__tests__/features/payments/payments-panel.test.tsx is excluded by !**/__tests__/** and included by packages/**
  • packages/studio/__tests__/lib/internal.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • plans/300-decode-doc-read-paths.md is excluded by none and included by none
  • plans/README.md is excluded by none and included by none
📒 Files selected for processing (4)
  • packages/shard-engine/src/introspect.ts
  • packages/studio/src/features/data/hooks/use-data-browser.tsx
  • packages/studio/src/features/payments/payments-panel.tsx
  • packages/studio/src/lib/internal.ts

Comment thread packages/shard-engine/src/introspect.ts Resolved
prisis and others added 2 commits August 4, 2026 00:07
Review catch, and a real regression from the decode. `!Array.isArray` was a
sufficient document check under bare `JSON.parse`, whose only non-plain root is
an array — but a `__doc__` whose ROOT is a tagged value parses as an array and
DECODES to a `Uint8Array` / `Date` / `Map`, which is neither null nor an array.
The old check waved those through:

    root ["$lunora.wire$","bytes","AAEC","ArrayBuffer"]
      → decodes to Uint8Array → spreads to {"0":1,"1":2,"2":3}

i.e. a page of junk numeric columns where the expansion should have bailed.

All three read/write sites now narrow by prototype (`null` or
`Object.prototype`), matching `encodeWire`'s own definition of a plain object so
the two ends agree on what a document is. The editor's save path gained the
check outright — it had none, so a free-text `[1,2]` or a root-level tag would
have been handed to the writer as a document.

Tests cover bytes / date / map roots on both sides and are demonstrated to fail
against the array-only check. The bare-array case passes either way and is kept
as the control that shows the new tests discriminate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019WfuWNN3cWJE6beZjng347
… one document guard

Thermos review found the plan this branch executes was wrong about its own
primary surface, and that the fix for it was a regression.

The data browser never showed a raw tag. `readTablePage`'s result goes back
through `jsonResponse`, and `LunoraClient` decodes the whole response
(`decodeWire(body.result)`), so a tagged array is already a real bigint by the
time `formatCell` runs. Decoding inside `introspect.ts` broke that: `jsonResponse`
is the one DO result path that does NOT `encodeWire`, so a decoded `v.bigint()`
made `JSON.stringify` throw — every bigint table would have returned a redacted
500 — and a decoded `v.bytes()` flattened to `{}` before the client saw it.

That decode is reverted. A test pins the invariant instead: the page must stay
JSON-serializable, and the tagged form must decode to `1000n` across a simulated
wire. It fails with the decode reinstated.

What stays, because it was never on that path:

- `rowDocument` / `readField` parse a raw `__doc__` STRING, which the client's
  decode leaves alone (a JSON string stays a string), so they must decode.
- The editor's encode-prefill / decode-save pair, which fixes a pre-existing
  defect: `JSON.stringify` throws on a bigint, so editing any row of a
  bigint-bearing table already failed.
- `formatCell`'s byte rendering — a decoded ArrayBuffer really does reach it.

Also from review:

- The "is this a document" predicate existed in five near-copies, already drifted
  (one rejected null-prototype objects, the others accepted them). It now lives
  once in `shared/wire-codec.ts` as `isPlainObject` + `decodeDocument`, next to
  the encoder that defines the term. Net deletion.
- `formatCell` reuses the `formatBytes` that already lived 70 lines below it, so
  a 2 MB blob reads `<bytes: 2.0 MB>` rather than `<bytes: 2097152>`.
- The editor reports a JSON syntax error and a malformed tag separately, instead
  of labelling a codec RangeError "Invalid JSON".
- Three comments corrected: a typed array stringifies to its indices, not `{}`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019WfuWNN3cWJE6beZjng347
…bigint

Pre-existing on alpha, surfaced by the review of this branch. `LunoraClient`
decodes the wire codec on the way in, so a `v.bigint()` column arrives in the
studio as a real `bigint` and `v.bytes()` as an `ArrayBuffer`. Three surfaces
then serialize a row without accounting for that:

- the JSON view (`data-browser-page.tsx`) — `JSON.stringify` throws during
  RENDER, so the whole Data page dies rather than just the toggle;
- "Export page → JSON" (`grid-features.tsx`) — throws inside the click handler,
  so the download silently never happens and nothing is surfaced;
- the `hash` mask strategy (`mask-preview.ts`) — throws and fails closed to the
  custom-mask sentinel, so a masked bigint column shows the sentinel instead of
  a hash.

Reachable today: `@lunora/payment`'s `paymentSessions` has three `v.bigint()`
columns, so opening it and switching to JSON view is enough. The CSV exporter
already handled bigint explicitly, which is how it went unnoticed in the others.

One shared `jsonRowReplacer` covers the two stringify sites, rendering a bigint
as a decimal string (JSON has no bigint) and bytes as the same `<bytes: N>`
summary a cell shows. `mask-preview` only special-cases bigint, so every other
value's hash is byte-for-byte unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019WfuWNN3cWJE6beZjng347
@codspeed

codspeed Bot commented Aug 3, 2026

Copy link
Copy Markdown

Merging this PR will regress 3 benchmarks

⚡ 1 improved benchmark
❌ 3 regressed benchmarks
✅ 249 untouched benchmarks
⏩ 10 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ bare: patch (single-field update on seed row) 611.3 µs 798.7 µs -23.47%
❌ 1 shard × 1000 rows (single round-trip) 2.7 ms 3.1 ms -11.67%
❌ flat 3 primitives (the notify.send attribute shape) 55.6 µs 62.2 µs -10.63%
⚡ broadcastWhisper to 128 members 508.3 µs 254.6 µs +99.63%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing advisor/300-decode-doc-read-paths (3c9975c) with alpha (ab0afaf)2

Open in CodSpeed

Footnotes

  1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on alpha (adecf8a) during the generation of this report, so ab0afaf was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@prisis
prisis merged commit 73698a3 into alpha Aug 4, 2026
46 of 47 checks passed
@prisis
prisis deleted the advisor/300-decode-doc-read-paths branch August 4, 2026 06:20
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — 3c9975ce Deployed Aug 3, 2026 by prisis via Benchmarks #1426
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant