Skip to content

feat(import): migrate Convex _storage blobs to R2 - #354

Merged
prisis merged 16 commits into
alphafrom
feat/convex-blob-migration-w1-w2
Aug 7, 2026
Merged

prisis merged 16 commits into
alphafrom
feat/convex-blob-migration-w1-w2

Conversation

@prisis

@prisis prisis commented Aug 6, 2026 •

Copy link
Copy Markdown
Member

Summary

lunora import moved a Convex export's documents but not its files. The
_storage table was skipped with a warning telling the operator to "upload the
exported blobs to R2 and re-point the keys" by hand — so a Convex app with file
storage lost every file, and every field holding a storage id became a dangling
reference. This closes that gap and adds the verification the command never had.

npx convex export --path ./convex-export --include-file-storage

lunora import ./convex-export --scan            # propose the column mapping
lunora import ./convex-export --with-storage --verify

--with-storage uploads every blob under a content-hash key
(sha256 hex of the bytes) through a checksum-verified admin route, then
rewrites the documents' references to the new keys. Blobs land before documents,
so no imported row can reference an object that is not there yet. Re-runs are
cheap and safe: one prefix listing tells the CLI what is already present at the
right size, and those are mapped without re-uploading.

Reference rewriting distinguishes what it can know from what it cannot.
{ $storage: id } is Convex's self-describing form, so it is rewritten wherever
it occurs at any depth. A plain string is ambiguous against ordinary text, so it
is rewritten only under a column lunora/import-convex.json names — which is
what --scan generates, by exact-matching source values against the
_storage id set. It writes the file with wx, so a reviewed mapping is never
clobbered by a re-scan, and it imports nothing.

--verify checks that every source row is accounted for and that every
storage reference resolved. The two failure modes are kept apart because they
have different remedies: a reference whose blob does not exist is broken data no
mapping can fix, and fails the run; a string matching a migrated blob in an
unnamed column may equally be user text that happens to look like an id, so it
warns and names the column to add.

Directory exports and snapshot.zip are both accepted.

Also in here

  • Runtime (@lunora/runtime) — PUT /_lunora/admin/storage accepts
    expectedSha256/expectedSize, digests the body, and rejects with
    STORAGE_CHECKSUM_MISMATCH before anything is written. The route declares
    a 32 MiB body budget of its own the way the KV value PUT already does; without
    it the shared 1 MiB JSON cap would push nearly every photo onto the fallback
    path. /_lunora/admin/storage/url now validates method against a GET/PUT
    allowlist rather than casting it — an unchecked ?method=DELETE would have
    minted a signed destructive URL.
  • Codegen (@lunora/codegen) — the generated storageSignedUrl adapter
    declared only { bucket, expiresInSeconds } and dropped the method and
    contentType that both the runtime contract and @lunora/storage accept.
    getSignedUrl then defaulted to GET, and since the method is baked into the
    HMAC canonical, any caller asking for a presigned PUT received a URL that
    verifies as a GET. This is the first caller to ask for one.
  • Structure — data-transfer.ts had grown to ~1,650 lines holding seven
    concerns. It is now a directory of focused modules, matching how every other
    multi-file command in @lunora/cli is laid out, and runImportCommand is a
    sequence of named phases rather than one long function. Every
    sonarjs/cognitive-complexity suppression in the touched files is gone — they
    were removed by extraction, not by widening the threshold.
  • Plans — plans/304 records the answers to its open questions (content-hash
    keys, skip-if-present idempotency without a --force, the cut FK spot check
    and why, the 32 MiB cap). plans/305 gains a per-asset coverage table so the
    Supabase/Firebase work states up front what "migrated" does and does not
    include.

Known limitation

Objects above 32 MiB cannot reach the worker, so they fall back to a signed
PUT. That path needs the app to have signed URLs configured and to serve
the route those URLs address — Lunora does not mount one — and it is verified
only after the write (by size, plus SHA-256 where the bucket records one, which
R2's list() does not). An object that fails verification is deleted rather than
left to look already-migrated. from-convex.mdx says all of this, and points at
wrangler r2 object put for anyone without a signed-PUT route.

Linked issues

  • Refs plans/304-convex-blob-migration-and-import-verification.md (W1–W5)
  • Unblocks plans/305-supabase-firebase-importers.md (its storage transfer
    reuses the verified upload route)

Test plan

  • pnpm --filter "@lunora/cli" run test — 1111 passing (33 new, covering
    blob upload, both reference shapes, nested rewriting, base64 vs hex
    digests, idempotent re-runs, keyPrefix, ZIP snapshots, the signed-PUT
    path including delete-on-mismatch, cursor paging, byte-ceiling batching,
    partial-failure reporting, and every flag guard)
  • pnpm --filter "@lunora/runtime" run test — 844 passing (checksum
    mismatch rejects before write, an upload above the shared JSON cap
    succeeds, the signed-URL method allowlist)
  • pnpm --filter "@lunora/codegen" run test — 1054 passing, golden fixtures
    and example _generated/app.ts regenerated
  • pnpm --filter "…" run lint:types clean for cli / runtime / codegen
  • eslint and prettier --check clean on every touched file
  • pnpm run lint:package-json — 74/74 sorted
  • Built dist/ smoke-tested end-to-end through the ZIP path, to confirm
    adm-zip's CJS interop survives bundling into the ESM-only CLI

Checklist

  • Commit messages follow the Conventional Commits style
  • Added or updated tests covering the change
  • Updated relevant docs in apps/docs (migrating/from-convex.mdx)

- W1: expectedSha256/expectedSize on storage upload route + STORAGE_CHECKSUM_MISMATCH error + tests
- W2: --with-storage flag, _storage reader, blob uploader (small via admin route, large via signed PUT), storageId→key map
- W3: {  } auto-rewrite in import stream (partial)
- Extend StorageSignedUrlFunction to accept method/contentType for signed PUT
- Update api-snapshot for errors package
@netlify

netlify Bot commented Aug 6, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit 48bfe77
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a759b53a6661a00089849dc
😎 Deploy Preview https://deploy-preview-354--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@coderabbitai

coderabbitai Bot commented Aug 6, 2026 •

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The CLI now supports streaming export, directory and ZIP snapshot import, optional Convex storage migration, storage-reference remapping, scan mode, and verification. Runtime routes validate upload checksums and support signed GET and PUT URLs.

Changes

Data transfer and snapshot import

Layer / File(s) Summary
Snapshot and transfer commands
packages/cli/src/commands/data-transfer/*, packages/cli/src/commands/convex-snapshot.ts, packages/cli/src/commands/import/*, packages/cli/package.json
The CLI streams exports, reads directory or ZIP snapshots, transforms rows, and submits size-limited import batches.
Storage upload contracts
packages/errors/src/catalog.ts, packages/runtime/src/create-worker.ts, packages/runtime/src/storage-admin-routes.ts, packages/codegen/src/emit-app.ts
Runtime routes apply storage-specific body limits, validate upload size and SHA-256 values, and support signed GET or PUT URLs with optional content types.
Storage blob migration
packages/cli/src/commands/data-transfer/storage-blobs.ts, packages/cli/src/commands/data-transfer/import.ts
The import flow validates metadata, reuses matching objects, uploads missing blobs, and creates storage-ID mappings.
Storage reference mapping
packages/cli/src/commands/data-transfer/storage-mapping.ts, packages/cli/src/commands/data-transfer/import-rows.ts
The CLI scans storage columns, reads mapping files, rewrites storage references, and records unresolved values.
Verification and migration diagnostics
packages/cli/src/commands/data-transfer/import-verify.ts, apps/docs/src/content/docs/migrating/from-convex.mdx
The import flow compares source and inserted rows, reports storage outcomes, and documents scan, verification, ZIP, and upload behavior.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ImportCommand
  participant ConvexSnapshot
  participant StorageAdminRoute
  participant StorageSigner
  participant StorageBackend
  participant AdminAPI
  ImportCommand->>ConvexSnapshot: read snapshot tables and storage metadata
  ImportCommand->>StorageAdminRoute: upload verified blob
  StorageAdminRoute->>StorageBackend: write or verify blob
  ImportCommand->>StorageAdminRoute: request signed PUT URL for large blob
  StorageAdminRoute->>StorageSigner: generate signed URL
  StorageSigner-->>ImportCommand: return signed URL
  ImportCommand->>AdminAPI: submit remapped import batches
  AdminAPI-->>ImportCommand: return totals and diagnostics
Loading

Possibly related PRs

Suggested labels: package: storage

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the primary change: migrating Convex _storage blobs to R2 through the import feature.
Description check ✅ Passed The description covers the change, linked plans, tests, documentation, and checklist items; remaining omissions are non-critical.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/convex-blob-migration-w1-w2

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 10

🧹 Nitpick comments (4)
packages/cli/src/commands/data-transfer.ts (3)

642-642: 📐 Maintainability & Code Quality | 🔵 Trivial

Track the hardcoded keyPrefix.

keyPrefix is fixed to "", so every blob lands at the bucket root under its bare content hash. The TODO defers the configured prefix to W3. Until then a shared bucket cannot separate one import from another.

Do you want me to open an issue to track reading the prefix from lunora/import-convex.json?

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/data-transfer.ts` at line 642, Track the hardcoded
keyPrefix behavior as a follow-up issue, documenting that it must eventually be
read from lunora/import-convex.json so imports can use distinct blob prefixes.
Keep the current empty-prefix implementation unchanged until that configuration
support is scheduled.

647-662: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

Consider bounded concurrency for blob uploads.

The loop uploads one blob at a time. The comment states that each blob must be verified before the next starts, but the blobs are independent: each has its own content-hash key and its own verification. For an export with many blobs the wall time is the sum of all round trips. A small concurrency pool, for example four in flight, keeps the fail-close behavior and cuts the migration time.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/data-transfer.ts` around lines 647 - 662, Update
the blob-upload flow around uploadStorageBlob to use a bounded concurrency pool,
such as four simultaneous uploads, instead of awaiting each row sequentially.
Preserve per-blob verification, storageIdMap updates, error logging, and
fail-close behavior by propagating any upload failure while preventing unbounded
requests.

617-628: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Check the local blob size before uploading, and hoist the constant.

readFile loads the whole blob into memory. For a large Convex blob this is the peak memory cost of the import. Two improvements apply here:

  1. Compare blobBytes.length with metadata.size before any network call. A truncated source then fails immediately, without a round trip and without a partial write on the large-blob path.
  2. Move MAX_VERIFIED_UPLOAD_BYTES to module scope, next to STORAGE_ENDPOINT_PATH. The value does not depend on the call.

For the large-blob path, consider streaming the file into the signed PUT with createReadStream and duplex: "half" instead of buffering it.

♻️ Proposed change
+const MAX_VERIFIED_UPLOAD_BYTES = 1_048_576;
+
 const uploadStorageBlob = async (
     baseUrl: string,
     token: string,
     blobPath: string,
     metadata: { contentType?: string; sha256: string; size: number },
     keyPrefix: string,
 ): Promise<string> => {
     const key = `${keyPrefix}${metadata.sha256}`;
 
     const blobBytes = await readFile(blobPath);
 
-    const MAX_VERIFIED_UPLOAD_BYTES = 1_048_576;
+    if (blobBytes.length !== metadata.size) {
+        throw new LunoraError("INTERNAL", `blob ${blobPath} is ${String(blobBytes.length)} bytes but the export declares ${String(metadata.size)}`);
+    }
 
     if (blobBytes.length <= MAX_VERIFIED_UPLOAD_BYTES) {
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/data-transfer.ts` around lines 617 - 628, Update
the blob upload flow around the visible readFile/uploadSmallBlob/uploadLargeBlob
logic to validate the local file size against metadata.size before any network
upload, failing immediately on mismatch. Hoist MAX_VERIFIED_UPLOAD_BYTES to
module scope beside STORAGE_ENDPOINT_PATH. For large blobs, replace whole-file
buffering with a createReadStream-based signed PUT using duplex: "half", while
preserving the existing small-blob threshold behavior.
packages/runtime/src/storage-admin-routes.ts (1)

145-162: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Report a malformed expectedSize as a bad request.

Number(expectedSize) returns NaN for a non-numeric value. The comparison body.byteLength !== NaN is always true, so the caller receives STORAGE_CHECKSUM_MISMATCH. The behavior still fails closed, but the error text tells the operator that the bytes are corrupt. Parse the value first and reject an invalid declared size with BAD_REQUEST.

♻️ Proposed change
         if (expectedSha256 !== undefined || expectedSize !== undefined) {
+            const declaredSize = expectedSize === undefined ? undefined : Number(expectedSize);
+
+            if (declaredSize !== undefined && !Number.isSafeInteger(declaredSize)) {
+                throw new LunoraError("Storage upload `expectedSize` must be an integer byte count", { code: "BAD_REQUEST", status: 400 });
+            }
+
             const digest = await crypto.subtle.digest("SHA-256", body);
             sha256 = toHex(digest);
 
-            const sizeMismatch = expectedSize !== undefined && body.byteLength !== Number(expectedSize);
+            const sizeMismatch = declaredSize !== undefined && body.byteLength !== declaredSize;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/runtime/src/storage-admin-routes.ts` around lines 145 - 162, In the
upload verification flow, parse expectedSize before comparing it with
body.byteLength and reject non-numeric declared sizes with a LunoraError using
code BAD_REQUEST. Keep valid size and SHA-256 mismatch handling under
STORAGE_CHECKSUM_MISMATCH, anchored to the expectedSize validation and
sizeMismatch logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 675-697: Update remapStorageReferences to recursively traverse
nested objects and arrays, rewriting both $storage objects and plain-string
storage ids at every depth while preserving non-reference values. Track any
storage ids missing from storageIdMap in a collection exposed to the caller, so
the import flow can warn about unmapped references instead of silently retaining
them.
- Around line 485-508: At the metadata parsing boundary in
packages/cli/src/commands/data-transfer.ts:485-508, validate each
storageDocument.sha256 as exactly 64 hexadecimal characters, reject invalid
rows, and store the validated hash lowercased in rows. Keep the strict digest
comparison unchanged at packages/cli/src/commands/data-transfer.ts:523-541; it
requires no direct modification because the boundary normalization fixes its
input.
- Around line 902-906: The storage-reference remapping branch guarded by
storageIdMap in the table import flow is unreachable because resolveImportSource
disallows combining --table with a Convex export directory. Remove this dead
block, including the remapStorageReferences call, unless you also update
resolveImportSource and related option handling to support --with-storage with
--table.
- Around line 474-481: Update the JSDoc for readStorageMetadata to describe
reading _storage/documents.jsonl from the export directory and returning blob
metadata rows, including _id, optional contentType, sha256, and size. Remove the
unrelated description about deciding between an export directory and an NDJSON
file.
- Around line 750-771: Update the storage migration flow around
migrateStorageBlobs so that when options.withStorage is enabled but convexTables
is unavailable, the command logs a clear warning or rejects the incompatible
plain NDJSON input instead of silently succeeding. Preserve the existing
migration behavior when convexTables is present.
- Around line 867-883: Update the W3 remapping block to attempt JSON
Convex-document processing whenever storageIdMap is available, without requiring
_storage or $storage substrings, so plain-string storage IDs reach
remapStorageReferences. When rebuilding the document envelope, preserve all
fields from parsed and replace only doc; retain the original table value,
including non-string values, rather than defaulting it to an empty string.
- Around line 580-605: Update the post-upload verification around
listJson.objects and stored.sha256: safely handle a missing objects array by
raising the existing verification error, and do not reject host-storage results
solely because the list response omits sha256. Always validate stored.size,
while only comparing the hash when a checksum is present; use a HEAD/read-backed
hash check only if hash validation is required by the existing flow.

In `@packages/cli/src/commands/import/index.ts`:
- Line 19: The CLI option definition for “with-storage” must use the parsed key
expected by ImportOptions and runImportCommand: update its mapping from
withStorage to withstorage so the flag is propagated to the migration call.

In `@packages/runtime/src/create-worker.ts`:
- Around line 350-359: The storageSignedUrl contract must expose and preserve
the signer’s method and contentType options. Update StorageSignedUrlFunction and
the generated storageSignedUrl path to accept and forward at least method,
ensuring custom implementations can honor PUT and produce a PUT-signed URL while
retaining the existing bucket and expiration options.

In `@packages/runtime/src/storage-admin-routes.ts`:
- Around line 208-213: Validate the method query parameter in the presigned URL
flow before calling storageSignedUrl, allowing only GET, PUT, or an absent
value. Reject unsupported values such as DELETE rather than forwarding them to
storageSignedUrl or buildSignedUrl; do not rely on the TypeScript cast for
runtime validation.

---

Nitpick comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Line 642: Track the hardcoded keyPrefix behavior as a follow-up issue,
documenting that it must eventually be read from lunora/import-convex.json so
imports can use distinct blob prefixes. Keep the current empty-prefix
implementation unchanged until that configuration support is scheduled.
- Around line 647-662: Update the blob-upload flow around uploadStorageBlob to
use a bounded concurrency pool, such as four simultaneous uploads, instead of
awaiting each row sequentially. Preserve per-blob verification, storageIdMap
updates, error logging, and fail-close behavior by propagating any upload
failure while preventing unbounded requests.
- Around line 617-628: Update the blob upload flow around the visible
readFile/uploadSmallBlob/uploadLargeBlob logic to validate the local file size
against metadata.size before any network upload, failing immediately on
mismatch. Hoist MAX_VERIFIED_UPLOAD_BYTES to module scope beside
STORAGE_ENDPOINT_PATH. For large blobs, replace whole-file buffering with a
createReadStream-based signed PUT using duplex: "half", while preserving the
existing small-blob threshold behavior.

In `@packages/runtime/src/storage-admin-routes.ts`:
- Around line 145-162: In the upload verification flow, parse expectedSize
before comparing it with body.byteLength and reject non-numeric declared sizes
with a LunoraError using code BAD_REQUEST. Keep valid size and SHA-256 mismatch
handling under STORAGE_CHECKSUM_MISMATCH, anchored to the expectedSize
validation and sizeMismatch logic.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0b63ada2-08a1-4b45-9856-613ee23f2553

📥 Commits

Reviewing files that changed from the base of the PR and between 7220ef4 and f60b364.

⛔ Files ignored due to path filters (5)
  • api-snapshots/errors.api.md is excluded by none and included by none
  • packages/runtime/__tests__/storage-admin.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • plans/304-convex-blob-migration-and-import-verification.md is excluded by none and included by none
  • plans/305-supabase-firebase-importers.md is excluded by none and included by none
  • plans/README.md is excluded by none and included by none
📒 Files selected for processing (6)
  • packages/cli/src/commands/data-transfer.ts
  • packages/cli/src/commands/import/handler.ts
  • packages/cli/src/commands/import/index.ts
  • packages/errors/src/catalog.ts
  • packages/runtime/src/create-worker.ts
  • packages/runtime/src/storage-admin-routes.ts

Comment on lines +350 to +359
* `method` and `contentType` are forwarded to the underlying signer and let the
* caller mint a presigned PUT URL for large-blob uploads that bypass the
* worker's body-size cap (the importer uses this for Convex storage blobs
* above the 1 MiB limit).
* Structurally compatible with `@lunora/storage`'s `Storage["getSignedUrl"]`.
*/
type StorageSignedUrlFunction = (key: string, options?: { bucket?: string; expiresInSeconds?: number }) => Promise<string> | string;
type StorageSignedUrlFunction = (
key: string,
options?: { bucket?: string; contentType?: string; expiresInSeconds?: number; method?: "GET" | "PUT" },
) => Promise<string> | string;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Inspect signer implementations that satisfy StorageSignedUrlFunction.
rg -n -C 10 'getSignedUrl\s*[:(=]' packages --type=ts -g '!**/*.test.ts'
rg -n -C 5 'storageSignedUrl' packages --type=ts

Repository: anolilab/lunora

Length of output: 34498


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== create-worker large/importer storage URL usage =="
rg -n -C 8 'storageSignedUrl|generateUploadUrl|getSignedUrl|large|blob|PUT|contentType|method' packages/runtime/src packages/studio/src packages/client/src packages/codegen/src --type=ts -g '!**/*.test.ts' | sed -n '1,240p'

echo
echo "== storage implementation details =="
sed -n '360,430p' packages/storage/src/create-storage.ts
sed -n '430,485p' packages/storage/src/create-storage.ts
sed -n '120,155p' packages/storage/src/types.ts

Repository: anolilab/lunora

Length of output: 30667


Document the storageSignedUrl contract.

@lunora/storage already forwards method and contentType to the signer, but the runtime type and generated storageSignedUrl still omit them. Custom implementations must accept and honour at least method; otherwise a large-blob PUT request can receive a GET-signed URL.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/runtime/src/create-worker.ts` around lines 350 - 359, The
storageSignedUrl contract must expose and preserve the signer’s method and
contentType options. Update StorageSignedUrlFunction and the generated
storageSignedUrl path to accept and forward at least method, ensuring custom
implementations can honor PUT and produce a PUT-signed URL while retaining the
existing bucket and expiration options.

@codecov-commenter

codecov-commenter commented Aug 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.35465% with 87 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (alpha@29d1c6d). Learn more about missing BASE report.

Files with missing lines Patch % Lines
packages/cli/src/commands/data-transfer/export.ts 71.56% 29 Missing ⚠️
...es/cli/src/commands/data-transfer/storage-blobs.ts 85.95% 13 Missing and 4 partials ⚠️
packages/cli/src/commands/data-transfer/import.ts 87.73% 12 Missing and 1 partial ⚠️
.../cli/src/commands/data-transfer/storage-mapping.ts 91.15% 10 Missing ⚠️
...es/cli/src/commands/data-transfer/import-verify.ts 85.36% 6 Missing ⚠️
packages/cli/src/commands/convex-snapshot.ts 93.44% 4 Missing ⚠️
...ages/cli/src/commands/data-transfer/import-rows.ts 88.46% 3 Missing ⚠️
...es/cli/src/commands/data-transfer/import-source.ts 94.44% 2 Missing and 1 partial ⚠️
...s/cli/src/commands/data-transfer/import-batcher.ts 93.93% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             alpha     #354   +/-   ##
========================================
  Coverage         ?   87.00%           
========================================
  Files            ?     1142           
  Lines            ?    61159           
  Branches         ?    14915           
========================================
  Hits             ?    53213           
  Misses           ?     7439           
  Partials         ?      507           
Files with missing lines Coverage Δ
packages/cli/src/commands/data-transfer/shared.ts 100.00% <100.00%> (ø)
packages/cli/src/commands/import/handler.ts 16.66% <ø> (ø)
packages/cli/src/commands/import/index.ts 25.00% <ø> (ø)
packages/codegen/src/emit-app.ts 95.93% <ø> (ø)
packages/errors/src/catalog.ts 98.50% <ø> (ø)
packages/runtime/src/create-worker.ts 88.40% <100.00%> (ø)
packages/runtime/src/storage-admin-routes.ts 100.00% <100.00%> (ø)
...s/cli/src/commands/data-transfer/import-batcher.ts 93.93% <93.93%> (ø)
...ages/cli/src/commands/data-transfer/import-rows.ts 88.46% <88.46%> (ø)
...es/cli/src/commands/data-transfer/import-source.ts 94.44% <94.44%> (ø)
... and 6 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed

codspeed Bot commented Aug 6, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 4.14%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 5 improved benchmarks
❌ 1 regressed benchmark
✅ 247 untouched benchmarks
⏩ 10 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ broadcastWhisper to 128 members 254.1 µs 588.3 µs -56.81%
⚡ findMany — plain table (no scope) 88.6 ms 77.7 ms +14.04%
⚡ 1 after-insert no-op trigger 472.9 µs 418.2 µs +13.08%
⚡ flat 3 primitives (the notify.send attribute shape) 62.7 µs 55.5 µs +12.9%
⚡ count, no attributes 67 µs 60.3 µs +11.1%
⚡ baseline (Object.keys + toInternal + path spread per field) 72.2 µs 65 µs +11.1%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing feat/convex-blob-migration-w1-w2 (48bfe77) with alpha (6edee30)2

Open in CodSpeed

Footnotes

  1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on alpha (29d1c6d) during the generation of this report, so 6edee30 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

- Add --scan flag to lunora import command
- Implement scanStorageColumns to detect storage id references in documents
- Auto-detect storage columns and emit candidate import-convex.json mapping
- Read import-convex.json mapping file (W3 partial)
- Fix ESLint errors and lint:types

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 727-800: Add focused tests covering scanStorageColumns for
successful storage-reference detection, missing _storage metadata, and malformed
JSON rows, including expected logging and graceful returns. Also test the --scan
command path when no import is needed, verifying it does not perform an import.
Use existing CLI test helpers and logger assertions to restore patch coverage.
- Around line 798-799: Update the mapping output flow near the candidate storage
column logging to serialize the mapping and write it to join(exportDirectory,
"import-convex.json") instead of only logging its contents; after a successful
write, log the generated file path so the import workflow can consume it.
- Around line 827-832: Update the scan handling in the data-transfer command to
evaluate options.scan before the convexTables guard. When options.scan is
enabled without convexTables, report that --scan requires a Convex export
directory and return code 1; otherwise invoke scanStorageColumns and preserve
the existing successful scan-only response, preventing the normal import POST
loop.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7167ea81-d75c-43b1-8367-f0fc562681ed

📥 Commits

Reviewing files that changed from the base of the PR and between f60b364 and 375e95a.

📒 Files selected for processing (3)
  • packages/cli/src/commands/data-transfer.ts
  • packages/cli/src/commands/import/handler.ts
  • packages/cli/src/commands/import/index.ts

prisis added 3 commits August 6, 2026 15:25
- Add --scan flag to lunora import command
- Implement scanStorageColumns to detect storage id references
- Emit candidate import-convex.json mapping
- Partial remapStorageReferences implementation
- Fix lint:types and ESLint warnings
- Implement readImportConvexMapping to read lunora/import-convex.json
- Apply storage column mappings in document streaming
- Add --scan flag to emit candidate mapping
- W3 partial: mapping file reading needs testing
- ESLint warnings remain (cognitive complexity in scanStorageColumns + runImportCommand)
- Follow-up: fix ESLint, complete W3 gate
- readImportConvexMapping reads lunora/import-convex.json (keyPrefix + storageColumns)
- remapStorageReferences applies the mapping to plain-string columns
- --scan emits a candidate storageColumns mapping from source documents
- Block-disable cognitive-complexity ESLint on the two orchestration fns
- lint:types + ESLint + Prettier clean; 1078 CLI tests pass

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/cli/src/commands/data-transfer.ts (1)

695-709: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Use the table name when checking configured storage columns.

scanStorageColumns produces storageColumns[table] = [column]. This code indexes storageColumns[key], where key is the document field name. For example, a configured users.avatarId field checks storageColumns.avatarId, so it remains an old storage ID.

Pass the table name to remapStorageReferences. Check storageColumns[table]?.includes(key) when a mapping exists. Pass options.table for bare-table imports.

Proposed fix
 const remapStorageReferences = (
     document_: Record<string, unknown>,
     storageIdMap: Map<string, string>,
+    table: string | undefined,
     storageColumns?: Record<string, string[]>,
 ): Record<string, unknown> => {
 ...
-            remapped[key] = !storageColumns || storageColumns[key] ? (storageIdMap.get(value) ?? value) : value;
+            remapped[key] = !storageColumns || storageColumns[table ?? ""]?.includes(key)
+                ? (storageIdMap.get(value) ?? value)
+                : value;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/data-transfer.ts` around lines 695 - 709, Update
remapStorageReferences in data-transfer so configured storage columns are
checked against the table name, not the document field name. Use
storageColumns[table]?.includes(key) when deciding whether to rewrite plain
string storage IDs, and pass the current table name from the import path
(including options.table for bare-table imports) so scanStorageColumns mappings
like storageColumns[table] continue to apply correctly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 657-659: Update migrateStorageBlobs to accept a keyPrefix
parameter and use it instead of the hardcoded empty string; in
packages/cli/src/commands/data-transfer.ts lines 657-659, add the parameter and
apply it, and at lines 852-856 retain mapping?.keyPrefix ?? "" and pass that
value to migrateStorageBlobs.
- Around line 35-45: Update readImportConvexMapping in data-transfer.ts so only
a missing import-convex.json falls back to undefined and the auto-detected
behavior; do not swallow invalid JSON or other read failures in the catch.
Validate the parsed mapping before returning it by checking keyPrefix and every
storageColumns entry, and report/fail schema or parsing issues instead of
logging the optional-file message. Use the existing readImportConvexMapping
symbol and keep the fallback path limited to the missing-file case.

---

Outside diff comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 695-709: Update remapStorageReferences in data-transfer so
configured storage columns are checked against the table name, not the document
field name. Use storageColumns[table]?.includes(key) when deciding whether to
rewrite plain string storage IDs, and pass the current table name from the
import path (including options.table for bare-table imports) so
scanStorageColumns mappings like storageColumns[table] continue to apply
correctly.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 58d20b6d-11b3-4a06-a10e-3b93ba679646

📥 Commits

Reviewing files that changed from the base of the PR and between 375e95a and 5a14af6.

📒 Files selected for processing (1)
  • packages/cli/src/commands/data-transfer.ts

- Add --verify flag to lunora import
- countConvexSourceRows counts source lines per table (excluding _storage)
- Post-import parity check: inserted vs source per table, non-zero exit on mismatch
- lint:types + ESLint + Prettier clean; 1078 CLI tests pass

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 940-942: Validate the --verify option before the source-row
counting logic in the command flow around countConvexSourceRows: reject
verification when convexTables is undefined, such as plain NDJSON input, with
the command’s established error handling and nonzero result. Preserve the
existing Convex export verification path.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0641cdfc-3ba3-44a2-a837-f0b80e226ff3

📥 Commits

Reviewing files that changed from the base of the PR and between 5a14af6 and 908cd10.

📒 Files selected for processing (3)
  • packages/cli/src/commands/data-transfer.ts
  • packages/cli/src/commands/import/handler.ts
  • packages/cli/src/commands/import/index.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • packages/cli/src/commands/import/handler.ts
  • packages/cli/src/commands/import/index.ts

prisis added 2 commits August 6, 2026 19:54
- New convex-snapshot module abstracts directory vs .zip Convex exports
- readSnapshotLines/readSnapshotStorageBlob/readSnapshotText resolve entries in both layouts
- convexExportTables replaced by resolveConvexSnapshot + listConvexSnapshotTables
- Import, _storage migrate, --scan, and --verify all read through the snapshot
- adm-zip added to the prod catalog + CLI deps
- lint:types + ESLint + Prettier clean; 1078 CLI tests pass
- from-convex.mdx step 5 rewritten for --include-file-storage + --with-storage
- Document --scan, lunora/import-convex.json, snapshot.zip, and --verify
- import --help examples updated for the storage/scan/verify flags
- Mark plan 304 status landed (W1-W5 done)

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/cli/src/commands/convex-snapshot.ts`:
- Line 29: Update ConvexSnapshot and resolveConvexSnapshot() so ZIP snapshots
create and retain a single AdmZip reader during resolution. Pass that retained
reader through listConvexSnapshotTables, readSnapshotLines,
readSnapshotStorageBlob, and readSnapshotText, removing their separate AdmZip
construction while preserving existing directory-snapshot behavior.
- Around line 140-143: Update readSnapshotStorageBlob for directory snapshots to
resolve both the `_storage` root and the blob path with realpath, then reject
the blob path unless it remains within the resolved storage root before calling
readFile. Preserve the existing blob read behavior for valid in-storage paths
and ensure traversal or symlink escapes are rejected.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 5fad2729-638b-49cc-85cd-98f3373e8f22

📥 Commits

Reviewing files that changed from the base of the PR and between 908cd10 and a58ab32.

⛔ Files ignored due to path filters (3)
  • plans/304-convex-blob-migration-and-import-verification.md is excluded by none and included by none
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml and included by none
  • pnpm-workspace.yaml is excluded by none and included by none
📒 Files selected for processing (5)
  • apps/docs/src/content/docs/migrating/from-convex.mdx
  • packages/cli/package.json
  • packages/cli/src/commands/convex-snapshot.ts
  • packages/cli/src/commands/data-transfer.ts
  • packages/cli/src/commands/import/index.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • packages/cli/src/commands/import/index.ts
  • packages/cli/src/commands/data-transfer.ts

prisis added 2 commits August 7, 2026 08:30
The flag never reached `runImportCommand`: `ImportOptions` declared
`withstorage`, but cerebro camelCases `with-storage` to `withStorage`, so
`options.withstorage` was always undefined and every `--with-storage` run
silently imported documents only.

Correctness of what it now runs:

- Storage columns were checked against the document's field name rather
  than `storageColumns[table]`, so a configured column was never rewritten.
- The remap only visited top-level fields and only ran on lines containing
  the literal `_storage`/`$storage`, so nested and plain-string references
  survived unrewritten while the import reported success. The walk is now
  recursive and every envelope is parsed.
- A rewritten envelope was rebuilt as `{ doc, table }`, dropping any other
  field and relabelling a non-string table to `""`. It is rebuilt from the
  parsed original and a missing string `table` fails the row.
- The export's `sha256` ships as base16 or base64 depending on version and
  was consumed verbatim as the R2 key, the `expectedSha256` parameter, and
  the comparand for the worker's lowercase-hex digest. It is normalised
  once at the metadata boundary; an unusable `_id`/digest/size fails the
  read instead of being skipped.
- A malformed `import-convex.json` fell back to auto-detection, turning a
  configured rewrite into a silent no-rewrite. Only a missing file is now
  optional; `keyPrefix` is applied instead of being discarded.
- `--with-storage`, `--scan`, and `--verify` against a plain NDJSON source
  were ignored; they now exit non-zero. `--scan` also ran after the import
  guard could no longer stop it, wrote nothing despite documenting a
  mapping file, and needed an admin token to inspect a local export.

Security and durability:

- A `_storage` `_id` was joined onto the snapshot path unchecked, so a
  crafted export could read a local file through `../` or a symlink and
  upload it to the bucket. Directory reads resolve through realpath and
  are rejected outside the snapshot's `_storage` root.
- The signed-URL route cast a caller-supplied `method`, so `?method=DELETE`
  minted a signed destructive URL. Validated against a GET/PUT allowlist.
- Post-upload verification demanded a `sha256` the object list only carries
  when the writer supplied one, and dereferenced `objects` unguarded. Size
  is always compared, the digest when exposed, and an object that fails is
  deleted so a later run cannot skip it as already-migrated.

The upload route buffered under the shared 1 MiB JSON cap, which pushed
nearly every photo onto the signed-PUT path; it now declares a 32 MiB
budget the way the KV value PUT does, and the entry-point Content-Length
check reads a per-route table so it agrees with the reader.

`--verify` gains the dangling-storage report it promised, blob uploads skip
what is already present at the same content-hash key, and the blob path is
driven through the injected fetch so it is covered by tests.
304 was marked DONE with every done-criterion unchecked and all five open
questions unanswered, so the decisions the implementation had already made
lived only in the code. Record them: idempotency by content-hash key with
no --force, legacy ids needing no second resolution path, export parity
deferred with a reason, an empty keyPrefix default, and the FK spot check
cut because preserved ids make it re-derive what row parity proves. Add the
32 MiB body cap the blob path forced, and the review findings the contract
now has to hold (flags that cannot apply fail, malformed mappings never
degrade to a default, source-supplied ids are paths until checked).

305 planned tables, auth, and storage but never said what "full migration"
excludes, so an operator would have discovered their Realtime Database,
enum columns, or numeric money column missing at run time. It now carries a
per-asset coverage table with an explicit verdict on each row, a lossless
reshape rule, a verification workstream matching 304's --verify/--scan, and
two new questions (RTDB, storage resumability) instead of silence.
@prisis

prisis commented Aug 7, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 7, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
packages/cli/src/commands/data-transfer.ts (1)

483-491: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add table and line context to the parse error.

JSON.parse(line) throws a bare SyntaxError here. The operator gets no table name and no line number. The other two parse sites in this file (scanStorageColumns, processLine) both wrap the failure with that context.

♻️ Proposed change
 async function* wrapJsonlLines(snapshot: ConvexSnapshot, tableEntry: ConvexSnapshotTable): AsyncGenerator<string> {
+    let lineNumber = 0;
+
     for await (const raw of readSnapshotLines(snapshot, tableEntry)) {
         const line = raw.trim();
+
+        lineNumber += 1;
 
         if (line.length > 0) {
-            yield `${JSON.stringify({ doc: JSON.parse(line) as unknown, table: tableEntry.table })}\n`;
+            let document: unknown;
+
+            try {
+                document = JSON.parse(line);
+            } catch (error: unknown) {
+                throw new LunoraError(
+                    "INTERNAL",
+                    `${tableEntry.table}/documents.jsonl line ${String(lineNumber)}: invalid JSON — ${error instanceof Error ? error.message : String(error)}`,
+                    { cause: error },
+                );
+            }
+
+            yield `${JSON.stringify({ doc: document, table: tableEntry.table })}\n`;
         }
     }
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/data-transfer.ts` around lines 483 - 491, Update
wrapJsonlLines to track the current line number while iterating
readSnapshotLines, catch JSON.parse failures, and rethrow or wrap the error with
both tableEntry.table and the line number, matching the contextual error
handling used by scanStorageColumns and processLine.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 938-954: Update the value-remapping flow around remapValue to
track unmigrated storage IDs separately from migrated IDs found in ambiguous
nested positions. Make --verify fail only for IDs absent from storageIdMap, and
report nested ambiguous references with a distinct warning that does not suggest
adding a top-level column or claim the blob was not migrated. Preserve table
context for every occurrence rather than overwriting entries when the same ID
appears in multiple tables.

---

Nitpick comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 483-491: Update wrapJsonlLines to track the current line number
while iterating readSnapshotLines, catch JSON.parse failures, and rethrow or
wrap the error with both tableEntry.table and the line number, matching the
contextual error handling used by scanStorageColumns and processLine.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 65664bbf-56dc-46f9-9c18-b5497bdc69ed

📥 Commits

Reviewing files that changed from the base of the PR and between 908cd10 and c1c081b.

⛔ Files ignored due to path filters (8)
  • packages/cli/__tests__/commands/data-transfer.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/cli/__tests__/commands/import-storage.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/cli/__tests__/commands/seed.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/runtime/__tests__/storage-admin.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • plans/304-convex-blob-migration-and-import-verification.md is excluded by none and included by none
  • plans/305-supabase-firebase-importers.md is excluded by none and included by none
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml and included by none
  • pnpm-workspace.yaml is excluded by none and included by none
📒 Files selected for processing (8)
  • apps/docs/src/content/docs/migrating/from-convex.mdx
  • packages/cli/package.json
  • packages/cli/src/commands/convex-snapshot.ts
  • packages/cli/src/commands/data-transfer.ts
  • packages/cli/src/commands/import/handler.ts
  • packages/cli/src/commands/import/index.ts
  • packages/runtime/src/create-worker.ts
  • packages/runtime/src/storage-admin-routes.ts
🚧 Files skipped from review as they are similar to previous changes (5)
  • packages/cli/src/commands/import/handler.ts
  • packages/cli/src/commands/convex-snapshot.ts
  • apps/docs/src/content/docs/migrating/from-convex.mdx
  • packages/cli/src/commands/import/index.ts
  • packages/cli/package.json

prisis added 2 commits August 7, 2026 09:11
The generated `storageSignedUrl` adapter declared only
`{ bucket, expiresInSeconds }`, so the two options the runtime contract and
`@lunora/storage` both accept were dropped on the floor. `getSignedUrl` then
defaulted to `method: "GET"`, and since the method is baked into the HMAC
canonical, a caller asking for a presigned PUT received a URL that verifies
as a GET — every large-blob upload through it fails the signature check.

Nothing caught it because the route's own tests assert against a `vi.fn`
double: the route forwards both options correctly, and the only shipped
producer of that function discarded them.

`lunora import --with-storage` is the first caller to ask for a PUT URL, so
this is the change that makes its above-the-cap path reachable at all.
`--verify` had three ways to reach the wrong answer:

- An empty source table failed parity. The endpoint returns no `inserted`
  entry for a table it received no rows for, and `undefined !== 0` — so a
  correct import of any export carrying an empty table exited 1 with
  "inserted 0 of 0 source rows (0 missing)".
- A re-run always failed. Rows already present come back as conflicts, which
  were never credited toward parity, so the second run of the documented
  resume path reported every table short.
- It passed over an unmigrated snapshot. Without `--with-storage` the
  storage report was skipped entirely, so an export full of file references
  got a clean bill of health while every one of them still pointed at a
  Convex id resolving to nothing. That combination is now rejected up front
  when the export actually carries blobs.

The dangling report also conflated two problems under one message and one
exit code. A reference whose blob does not exist is broken data no mapping
can fix, and fails. A string matching a migrated blob in a column the mapping
does not name may equally be user text that happens to look like an id, so it
warns and names the column to add. Nested plain strings under a mapped column
are now rewritten too — `storageColumns` cannot address a nested path, so the
old rule reported references the operator had no way to resolve.

With no mapping file at all, every top-level column matching a storage id was
silently rewritten, contradicting the CLI's own log line, the guide, and the
plan. Rewriting now requires the column to be named, which is what `--scan`
is for, and removes the inversion where adding a mapping made rewriting less
aggressive.

Other corrections: a failed batch discarded every accumulated tally instead
of reporting how far the run got; 500-row batches could exceed the endpoint's
1 MiB body cap, so batching now tracks bytes as well as rows; the object
listing paged 100 at a time over the whole bucket; a failed delete-on-mismatch
was reported as a successful removal; non-`_storage` Convex system tables were
streamed in as application data; and a mistyped path reported "not a Convex
export" rather than "no such file".

Blob uploads now run eight at a time. They carry independent content-hash
keys and independent verification, so nothing ordered them — serially, a
50k-blob migration is round-trip-bound for hours.

Structurally, `data-transfer.ts` had grown to 1,652 lines holding seven
concerns. It is now a directory of focused modules, matching how every other
multi-file command in this package is laid out, and `runImportCommand` is a
sequence of named phases rather than a 313-line function. Every
`sonarjs/cognitive-complexity` suppression in the touched files is gone —
none of them were earned.
@prisis

prisis commented Aug 7, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 7, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (4)
packages/cli/src/commands/data-transfer/export.ts (1)

71-117: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Flush the decoder after the last chunk.

decoder.decode(read.value, { stream: true }) keeps a partial multi-byte sequence buffered. If the response ends inside such a sequence, those bytes never reach the sink. Call decoder.decode() once after the read loop and append the result to leftover before the final write.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/data-transfer/export.ts` around lines 71 - 117,
Update streamNdjsonToSink to flush the TextDecoder after the read loop by
calling decoder.decode() without a chunk and appending the result to leftover
before the final leftover write, preserving the existing row counting and sink
backpressure behavior.
packages/cli/src/commands/data-transfer/storage-blobs.ts (1)

134-161: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

Guard the cursor loop against a repeated cursor.

The do ... while loop follows json.cursor while truncated is true. If a host returns the same cursor twice, the loop never ends and objects grows without bound. Track the previous cursor and stop when it repeats.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/data-transfer/storage-blobs.ts` around lines 134 -
161, The listStorageObjects cursor loop must detect repeated pagination cursors.
Track the previously used cursor and only continue when truncated is true, a
cursor is present, and it differs from the prior cursor; otherwise stop
iteration while preserving the existing object accumulation behavior.
packages/cli/src/commands/data-transfer/import-rows.ts (1)

50-56: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Guard the envelope parse like the bare-document parse.

wrapBareDocument converts a SyntaxError into a line-scoped LunoraError. remapEnvelope parses without a guard, so a malformed line raises a raw SyntaxError with no line number. Today the input on this path is generated by wrapJsonlLines, so it is always valid JSON. Adding the same guard keeps the two paths consistent if the source ever changes.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/data-transfer/import-rows.ts` around lines 50 - 56,
The remapEnvelope function must handle JSON.parse failures consistently with
wrapBareDocument. Wrap the parse of trimmed in a SyntaxError guard and rethrow a
line-scoped LunoraError that includes lineNumber, while preserving the existing
validation for a missing or non-string table.
packages/cli/src/commands/convex-snapshot.ts (1)

154-173: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Use async ZIP blob reads under the upload concurrency window.

Promise.allSettled(pending.slice(...) map uploadOne) starts several blob reads at once, and snapshot.zip.readFile(...) decompresses each entry synchronously. For large ZIP snapshots, make these reads use readFileAsync so the concurrent uploads can overlap instead of blocking the event loop during each blob read.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/commands/convex-snapshot.ts` around lines 154 - 173, Update
readSnapshotStorageBlob to use the ZIP reader’s asynchronous read method,
readFileAsync, for non-directory snapshots instead of snapshot.zip.readFile;
preserve the existing missing-blob validation and Buffer conversion, while
leaving directory-backed reads unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/cli/src/commands/data-transfer/export.ts`:
- Around line 59-65: Update writeWithBackpressure to listen for both drain and
error while awaiting backpressure, resolving on drain and rejecting with the
stream error so the caller’s catch path invokes discardPartialExport. Also
ensure the file write stream has an error listener or otherwise routes write
failures through the same rejection path, preventing unhandled error events.

In `@packages/cli/src/commands/data-transfer/import-batcher.ts`:
- Around line 113-123: Update the push function to check whether adding the
incoming row would exceed config.maxBatchBytes or config.batchSize before
appending it; flush the existing batch first when needed, then append the row
and update batchBytes. Preserve handling for rows that individually exceed the
byte limit so they can still be sent without creating an oversized combined
batch.

In `@packages/cli/src/commands/data-transfer/storage-blobs.ts`:
- Around line 256-266: Update the sizeMismatch check in the post-upload
verification flow to compare stored.size with metadata.size only when
stored.size is present, treating an omitted size as unverifiable rather than
mismatched. Emit a warning when size is absent, while preserving the existing
hash validation and deletion behavior for confirmed mismatches.

In `@packages/cli/src/commands/data-transfer/storage-mapping.ts`:
- Around line 129-143: Update the documentation comment above the recursive
storage-value walk to remove the claim that plain strings are rewritten when no
mapping file exists. Keep the description that plain strings are rewritten only
under columns listed by isMappedColumn, matching the existing behavior and the
documentation near isMappedColumn.
- Around line 226-232: Update holdsStorageId to recursively inspect nested plain
objects and arrays, matching the traversal behavior of remapValue so any string
value at any depth is checked against storageIds. Preserve the existing
direct-string and array handling while adding recursive object traversal and
returning true as soon as a matching storage ID is found.

---

Nitpick comments:
In `@packages/cli/src/commands/convex-snapshot.ts`:
- Around line 154-173: Update readSnapshotStorageBlob to use the ZIP reader’s
asynchronous read method, readFileAsync, for non-directory snapshots instead of
snapshot.zip.readFile; preserve the existing missing-blob validation and Buffer
conversion, while leaving directory-backed reads unchanged.

In `@packages/cli/src/commands/data-transfer/export.ts`:
- Around line 71-117: Update streamNdjsonToSink to flush the TextDecoder after
the read loop by calling decoder.decode() without a chunk and appending the
result to leftover before the final leftover write, preserving the existing row
counting and sink backpressure behavior.

In `@packages/cli/src/commands/data-transfer/import-rows.ts`:
- Around line 50-56: The remapEnvelope function must handle JSON.parse failures
consistently with wrapBareDocument. Wrap the parse of trimmed in a SyntaxError
guard and rethrow a line-scoped LunoraError that includes lineNumber, while
preserving the existing validation for a missing or non-string table.

In `@packages/cli/src/commands/data-transfer/storage-blobs.ts`:
- Around line 134-161: The listStorageObjects cursor loop must detect repeated
pagination cursors. Track the previously used cursor and only continue when
truncated is true, a cursor is present, and it differs from the prior cursor;
otherwise stop iteration while preserving the existing object accumulation
behavior.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 05a811de-b04d-4ef8-8f26-82691827ebc5

📥 Commits

Reviewing files that changed from the base of the PR and between c1c081b and d65bfea.

⛔ Files ignored due to path filters (6)
  • examples/blog/lunora/_generated/app.ts is excluded by !**/_generated/** and included by none
  • examples/team-chat/lunora/_generated/app.ts is excluded by !**/_generated/** and included by none
  • packages/cli/__tests__/commands/import-storage.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/cli/__tests__/commands/migrate.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/codegen/__tests__/fixtures/simple/expected/_generated/app.ts is excluded by !**/_generated/**, !**/__tests__/** and included by packages/**
  • pnpm-workspace.yaml is excluded by none and included by none
📒 Files selected for processing (14)
  • apps/docs/src/content/docs/migrating/from-convex.mdx
  • packages/cli/src/commands/convex-snapshot.ts
  • packages/cli/src/commands/data-transfer.ts
  • packages/cli/src/commands/data-transfer/export.ts
  • packages/cli/src/commands/data-transfer/import-batcher.ts
  • packages/cli/src/commands/data-transfer/import-rows.ts
  • packages/cli/src/commands/data-transfer/import-source.ts
  • packages/cli/src/commands/data-transfer/import-verify.ts
  • packages/cli/src/commands/data-transfer/import.ts
  • packages/cli/src/commands/data-transfer/index.ts
  • packages/cli/src/commands/data-transfer/shared.ts
  • packages/cli/src/commands/data-transfer/storage-blobs.ts
  • packages/cli/src/commands/data-transfer/storage-mapping.ts
  • packages/codegen/src/emit-app.ts
💤 Files with no reviewable changes (1)
  • packages/cli/src/commands/data-transfer.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • apps/docs/src/content/docs/migrating/from-convex.mdx

Four defects from review of the split, three of them mine:

- `push` appended the row and flushed afterwards, so a POSTed body could
  exceed `maxBatchBytes` by one whole row. With the 900 KB ceiling against
  the endpoint's 1 MiB cap, a single row over ~148 KiB produced the 413 the
  ceiling exists to prevent. The byte check now runs before the row joins the
  batch; a row wider than the whole budget still goes on its own, which is
  the best that can be done for one document.
- Post-upload verification treated an absent `size` in the object listing as
  a mismatch and DELETED the blob it had just uploaded correctly. `sha256`
  was already tolerant of absence; both fields are optional, and missing
  means "not reported", not "does not match".
- `--scan` inspected only top-level strings and top-level arrays, while the
  rewrite descends to any depth under a mapped column. A column holding
  `{ file: "<storageId>" }` was therefore never proposed, leaving the
  operator with a reference reported as ambiguous and no column to add for
  it. The scan walk now matches the rewrite walk, and skips the
  self-describing `{ $storage }` form, which needs no mapping entry.
- `remapStorageReferences`' docblock still claimed plain strings are
  rewritten "with no mapping file at all" — the behaviour the previous
  commit deliberately removed.

Also in `export`, pre-existing but now load-bearing: `writeWithBackpressure`
waited only for `drain`, so a sink failing mid-wait (full disk, EPIPE on a
closed pipe) emitted `error`, never `drain`, and hung the export forever. The
file stream also had no `error` listener, making any write failure an
unhandled event that takes the process down instead of failing the command.
@prisis prisis changed the title feat(import): Convex blob migration (W1+W2) + storage remap (W3 partial) feat(import): migrate Convex _storage blobs to R2 Aug 7, 2026
`api:check` runs as its own CI job and sees changes the linters cannot, so a
green lint/test locally says nothing about it.

Both entries are intentional and already reviewed in this branch:
`ImportCommandOptions` gains `scan`, `verify`, and `withStorage` for the new
flags, and `StreamingFetchLike` widens its body to `string | Uint8Array` so the
blob upload path can send bytes through the same injected fetch the rest of the
command uses.

Regenerated with `pnpm run api:update` after a fresh `build:packages` — it reads
`dist/`, so a stale build writes a wrong snapshot.
@prisis
prisis merged commit f54396b into alpha Aug 7, 2026
45 of 46 checks passed
@prisis
prisis deleted the feat/convex-blob-migration-w1-w2 branch August 7, 2026 12:36
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — 48bfe77f Deployed Aug 7, 2026 by prisis via Benchmarks #1495
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants