Repository navigation
feat(import): migrate Convex _storage blobs to R2 - #354
Conversation
- W1: expectedSha256/expectedSize on storage upload route + STORAGE_CHECKSUM_MISMATCH error + tests
- W2: --with-storage flag, _storage reader, blob uploader (small via admin route, large via signed PUT), storageId→key map
- W3: { } auto-rewrite in import stream (partial)
- Extend StorageSignedUrlFunction to accept method/contentType for signed PUT
- Update api-snapshot for errors package
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Thank you for following the naming conventions! 🙏 |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe CLI now supports streaming export, directory and ZIP snapshot import, optional Convex storage migration, storage-reference remapping, scan mode, and verification. Runtime routes validate upload checksums and support signed GET and PUT URLs. ChangesData transfer and snapshot import
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant ImportCommand
participant ConvexSnapshot
participant StorageAdminRoute
participant StorageSigner
participant StorageBackend
participant AdminAPI
ImportCommand->>ConvexSnapshot: read snapshot tables and storage metadata
ImportCommand->>StorageAdminRoute: upload verified blob
StorageAdminRoute->>StorageBackend: write or verify blob
ImportCommand->>StorageAdminRoute: request signed PUT URL for large blob
StorageAdminRoute->>StorageSigner: generate signed URL
StorageSigner-->>ImportCommand: return signed URL
ImportCommand->>AdminAPI: submit remapped import batches
AdminAPI-->>ImportCommand: return totals and diagnostics
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
There was a problem hiding this comment.
Actionable comments posted: 10
🧹 Nitpick comments (4)
packages/cli/src/commands/data-transfer.ts (3)
642-642: 📐 Maintainability & Code Quality | 🔵 TrivialTrack the hardcoded
keyPrefix.
keyPrefixis fixed to"", so every blob lands at the bucket root under its bare content hash. The TODO defers the configured prefix to W3. Until then a shared bucket cannot separate one import from another.Do you want me to open an issue to track reading the prefix from
lunora/import-convex.json?🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer.ts` at line 642, Track the hardcoded keyPrefix behavior as a follow-up issue, documenting that it must eventually be read from lunora/import-convex.json so imports can use distinct blob prefixes. Keep the current empty-prefix implementation unchanged until that configuration support is scheduled.
647-662: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy liftConsider bounded concurrency for blob uploads.
The loop uploads one blob at a time. The comment states that each blob must be verified before the next starts, but the blobs are independent: each has its own content-hash key and its own verification. For an export with many blobs the wall time is the sum of all round trips. A small concurrency pool, for example four in flight, keeps the fail-close behavior and cuts the migration time.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer.ts` around lines 647 - 662, Update the blob-upload flow around uploadStorageBlob to use a bounded concurrency pool, such as four simultaneous uploads, instead of awaiting each row sequentially. Preserve per-blob verification, storageIdMap updates, error logging, and fail-close behavior by propagating any upload failure while preventing unbounded requests.
617-628: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winCheck the local blob size before uploading, and hoist the constant.
readFileloads the whole blob into memory. For a large Convex blob this is the peak memory cost of the import. Two improvements apply here:
- Compare
blobBytes.lengthwithmetadata.sizebefore any network call. A truncated source then fails immediately, without a round trip and without a partial write on the large-blob path.- Move
MAX_VERIFIED_UPLOAD_BYTESto module scope, next toSTORAGE_ENDPOINT_PATH. The value does not depend on the call.For the large-blob path, consider streaming the file into the signed
PUTwithcreateReadStreamandduplex: "half"instead of buffering it.♻️ Proposed change
+const MAX_VERIFIED_UPLOAD_BYTES = 1_048_576; + const uploadStorageBlob = async ( baseUrl: string, token: string, blobPath: string, metadata: { contentType?: string; sha256: string; size: number }, keyPrefix: string, ): Promise<string> => { const key = `${keyPrefix}${metadata.sha256}`; const blobBytes = await readFile(blobPath); - const MAX_VERIFIED_UPLOAD_BYTES = 1_048_576; + if (blobBytes.length !== metadata.size) { + throw new LunoraError("INTERNAL", `blob ${blobPath} is ${String(blobBytes.length)} bytes but the export declares ${String(metadata.size)}`); + } if (blobBytes.length <= MAX_VERIFIED_UPLOAD_BYTES) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer.ts` around lines 617 - 628, Update the blob upload flow around the visible readFile/uploadSmallBlob/uploadLargeBlob logic to validate the local file size against metadata.size before any network upload, failing immediately on mismatch. Hoist MAX_VERIFIED_UPLOAD_BYTES to module scope beside STORAGE_ENDPOINT_PATH. For large blobs, replace whole-file buffering with a createReadStream-based signed PUT using duplex: "half", while preserving the existing small-blob threshold behavior.packages/runtime/src/storage-admin-routes.ts (1)
145-162: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winReport a malformed
expectedSizeas a bad request.
Number(expectedSize)returnsNaNfor a non-numeric value. The comparisonbody.byteLength !== NaNis always true, so the caller receivesSTORAGE_CHECKSUM_MISMATCH. The behavior still fails closed, but the error text tells the operator that the bytes are corrupt. Parse the value first and reject an invalid declared size withBAD_REQUEST.♻️ Proposed change
if (expectedSha256 !== undefined || expectedSize !== undefined) { + const declaredSize = expectedSize === undefined ? undefined : Number(expectedSize); + + if (declaredSize !== undefined && !Number.isSafeInteger(declaredSize)) { + throw new LunoraError("Storage upload `expectedSize` must be an integer byte count", { code: "BAD_REQUEST", status: 400 }); + } + const digest = await crypto.subtle.digest("SHA-256", body); sha256 = toHex(digest); - const sizeMismatch = expectedSize !== undefined && body.byteLength !== Number(expectedSize); + const sizeMismatch = declaredSize !== undefined && body.byteLength !== declaredSize;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/runtime/src/storage-admin-routes.ts` around lines 145 - 162, In the upload verification flow, parse expectedSize before comparing it with body.byteLength and reject non-numeric declared sizes with a LunoraError using code BAD_REQUEST. Keep valid size and SHA-256 mismatch handling under STORAGE_CHECKSUM_MISMATCH, anchored to the expectedSize validation and sizeMismatch logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 675-697: Update remapStorageReferences to recursively traverse
nested objects and arrays, rewriting both $storage objects and plain-string
storage ids at every depth while preserving non-reference values. Track any
storage ids missing from storageIdMap in a collection exposed to the caller, so
the import flow can warn about unmapped references instead of silently retaining
them.
- Around line 485-508: At the metadata parsing boundary in
packages/cli/src/commands/data-transfer.ts:485-508, validate each
storageDocument.sha256 as exactly 64 hexadecimal characters, reject invalid
rows, and store the validated hash lowercased in rows. Keep the strict digest
comparison unchanged at packages/cli/src/commands/data-transfer.ts:523-541; it
requires no direct modification because the boundary normalization fixes its
input.
- Around line 902-906: The storage-reference remapping branch guarded by
storageIdMap in the table import flow is unreachable because resolveImportSource
disallows combining --table with a Convex export directory. Remove this dead
block, including the remapStorageReferences call, unless you also update
resolveImportSource and related option handling to support --with-storage with
--table.
- Around line 474-481: Update the JSDoc for readStorageMetadata to describe
reading _storage/documents.jsonl from the export directory and returning blob
metadata rows, including _id, optional contentType, sha256, and size. Remove the
unrelated description about deciding between an export directory and an NDJSON
file.
- Around line 750-771: Update the storage migration flow around
migrateStorageBlobs so that when options.withStorage is enabled but convexTables
is unavailable, the command logs a clear warning or rejects the incompatible
plain NDJSON input instead of silently succeeding. Preserve the existing
migration behavior when convexTables is present.
- Around line 867-883: Update the W3 remapping block to attempt JSON
Convex-document processing whenever storageIdMap is available, without requiring
_storage or $storage substrings, so plain-string storage IDs reach
remapStorageReferences. When rebuilding the document envelope, preserve all
fields from parsed and replace only doc; retain the original table value,
including non-string values, rather than defaulting it to an empty string.
- Around line 580-605: Update the post-upload verification around
listJson.objects and stored.sha256: safely handle a missing objects array by
raising the existing verification error, and do not reject host-storage results
solely because the list response omits sha256. Always validate stored.size,
while only comparing the hash when a checksum is present; use a HEAD/read-backed
hash check only if hash validation is required by the existing flow.
In `@packages/cli/src/commands/import/index.ts`:
- Line 19: The CLI option definition for “with-storage” must use the parsed key
expected by ImportOptions and runImportCommand: update its mapping from
withStorage to withstorage so the flag is propagated to the migration call.
In `@packages/runtime/src/create-worker.ts`:
- Around line 350-359: The storageSignedUrl contract must expose and preserve
the signer’s method and contentType options. Update StorageSignedUrlFunction and
the generated storageSignedUrl path to accept and forward at least method,
ensuring custom implementations can honor PUT and produce a PUT-signed URL while
retaining the existing bucket and expiration options.
In `@packages/runtime/src/storage-admin-routes.ts`:
- Around line 208-213: Validate the method query parameter in the presigned URL
flow before calling storageSignedUrl, allowing only GET, PUT, or an absent
value. Reject unsupported values such as DELETE rather than forwarding them to
storageSignedUrl or buildSignedUrl; do not rely on the TypeScript cast for
runtime validation.
---
Nitpick comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Line 642: Track the hardcoded keyPrefix behavior as a follow-up issue,
documenting that it must eventually be read from lunora/import-convex.json so
imports can use distinct blob prefixes. Keep the current empty-prefix
implementation unchanged until that configuration support is scheduled.
- Around line 647-662: Update the blob-upload flow around uploadStorageBlob to
use a bounded concurrency pool, such as four simultaneous uploads, instead of
awaiting each row sequentially. Preserve per-blob verification, storageIdMap
updates, error logging, and fail-close behavior by propagating any upload
failure while preventing unbounded requests.
- Around line 617-628: Update the blob upload flow around the visible
readFile/uploadSmallBlob/uploadLargeBlob logic to validate the local file size
against metadata.size before any network upload, failing immediately on
mismatch. Hoist MAX_VERIFIED_UPLOAD_BYTES to module scope beside
STORAGE_ENDPOINT_PATH. For large blobs, replace whole-file buffering with a
createReadStream-based signed PUT using duplex: "half", while preserving the
existing small-blob threshold behavior.
In `@packages/runtime/src/storage-admin-routes.ts`:
- Around line 145-162: In the upload verification flow, parse expectedSize
before comparing it with body.byteLength and reject non-numeric declared sizes
with a LunoraError using code BAD_REQUEST. Keep valid size and SHA-256 mismatch
handling under STORAGE_CHECKSUM_MISMATCH, anchored to the expectedSize
validation and sizeMismatch logic.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0b63ada2-08a1-4b45-9856-613ee23f2553
⛔ Files ignored due to path filters (5)
api-snapshots/errors.api.mdis excluded by none and included by nonepackages/runtime/__tests__/storage-admin.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**plans/304-convex-blob-migration-and-import-verification.mdis excluded by none and included by noneplans/305-supabase-firebase-importers.mdis excluded by none and included by noneplans/README.mdis excluded by none and included by none
📒 Files selected for processing (6)
packages/cli/src/commands/data-transfer.tspackages/cli/src/commands/import/handler.tspackages/cli/src/commands/import/index.tspackages/errors/src/catalog.tspackages/runtime/src/create-worker.tspackages/runtime/src/storage-admin-routes.ts
| * `method` and `contentType` are forwarded to the underlying signer and let the | ||
| * caller mint a presigned PUT URL for large-blob uploads that bypass the | ||
| * worker's body-size cap (the importer uses this for Convex storage blobs | ||
| * above the 1 MiB limit). | ||
| * Structurally compatible with `@lunora/storage`'s `Storage["getSignedUrl"]`. | ||
| */ | ||
| type StorageSignedUrlFunction = (key: string, options?: { bucket?: string; expiresInSeconds?: number }) => Promise<string> | string; | ||
| type StorageSignedUrlFunction = ( | ||
| key: string, | ||
| options?: { bucket?: string; contentType?: string; expiresInSeconds?: number; method?: "GET" | "PUT" }, | ||
| ) => Promise<string> | string; |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect signer implementations that satisfy StorageSignedUrlFunction.
rg -n -C 10 'getSignedUrl\s*[:(=]' packages --type=ts -g '!**/*.test.ts'
rg -n -C 5 'storageSignedUrl' packages --type=tsRepository: anolilab/lunora
Length of output: 34498
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== create-worker large/importer storage URL usage =="
rg -n -C 8 'storageSignedUrl|generateUploadUrl|getSignedUrl|large|blob|PUT|contentType|method' packages/runtime/src packages/studio/src packages/client/src packages/codegen/src --type=ts -g '!**/*.test.ts' | sed -n '1,240p'
echo
echo "== storage implementation details =="
sed -n '360,430p' packages/storage/src/create-storage.ts
sed -n '430,485p' packages/storage/src/create-storage.ts
sed -n '120,155p' packages/storage/src/types.tsRepository: anolilab/lunora
Length of output: 30667
Document the storageSignedUrl contract.
@lunora/storage already forwards method and contentType to the signer, but the runtime type and generated storageSignedUrl still omit them. Custom implementations must accept and honour at least method; otherwise a large-blob PUT request can receive a GET-signed URL.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/runtime/src/create-worker.ts` around lines 350 - 359, The
storageSignedUrl contract must expose and preserve the signer’s method and
contentType options. Update StorageSignedUrlFunction and the generated
storageSignedUrl path to accept and forward at least method, ensuring custom
implementations can honor PUT and produce a PUT-signed URL while retaining the
existing bucket and expiration options.
Merging this PR will degrade performance by 4.14%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | broadcastWhisper to 128 members |
254.1 µs | 588.3 µs | -56.81% |
| ⚡ | findMany — plain table (no scope) |
88.6 ms | 77.7 ms | +14.04% |
| ⚡ | 1 after-insert no-op trigger |
472.9 µs | 418.2 µs | +13.08% |
| ⚡ | flat 3 primitives (the notify.send attribute shape) |
62.7 µs | 55.5 µs | +12.9% |
| ⚡ | count, no attributes |
67 µs | 60.3 µs | +11.1% |
| ⚡ | baseline (Object.keys + toInternal + path spread per field) |
72.2 µs | 65 µs | +11.1% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing feat/convex-blob-migration-w1-w2 (48bfe77) with alpha (6edee30)2
Footnotes
-
10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
alpha(29d1c6d) during the generation of this report, so 6edee30 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
- Add --scan flag to lunora import command - Implement scanStorageColumns to detect storage id references in documents - Auto-detect storage columns and emit candidate import-convex.json mapping - Read import-convex.json mapping file (W3 partial) - Fix ESLint errors and lint:types
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 727-800: Add focused tests covering scanStorageColumns for
successful storage-reference detection, missing _storage metadata, and malformed
JSON rows, including expected logging and graceful returns. Also test the --scan
command path when no import is needed, verifying it does not perform an import.
Use existing CLI test helpers and logger assertions to restore patch coverage.
- Around line 798-799: Update the mapping output flow near the candidate storage
column logging to serialize the mapping and write it to join(exportDirectory,
"import-convex.json") instead of only logging its contents; after a successful
write, log the generated file path so the import workflow can consume it.
- Around line 827-832: Update the scan handling in the data-transfer command to
evaluate options.scan before the convexTables guard. When options.scan is
enabled without convexTables, report that --scan requires a Convex export
directory and return code 1; otherwise invoke scanStorageColumns and preserve
the existing successful scan-only response, preventing the normal import POST
loop.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 7167ea81-d75c-43b1-8367-f0fc562681ed
📒 Files selected for processing (3)
packages/cli/src/commands/data-transfer.tspackages/cli/src/commands/import/handler.tspackages/cli/src/commands/import/index.ts
- Add --scan flag to lunora import command - Implement scanStorageColumns to detect storage id references - Emit candidate import-convex.json mapping - Partial remapStorageReferences implementation - Fix lint:types and ESLint warnings
- Implement readImportConvexMapping to read lunora/import-convex.json - Apply storage column mappings in document streaming - Add --scan flag to emit candidate mapping - W3 partial: mapping file reading needs testing - ESLint warnings remain (cognitive complexity in scanStorageColumns + runImportCommand) - Follow-up: fix ESLint, complete W3 gate
- readImportConvexMapping reads lunora/import-convex.json (keyPrefix + storageColumns) - remapStorageReferences applies the mapping to plain-string columns - --scan emits a candidate storageColumns mapping from source documents - Block-disable cognitive-complexity ESLint on the two orchestration fns - lint:types + ESLint + Prettier clean; 1078 CLI tests pass
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/cli/src/commands/data-transfer.ts (1)
695-709: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winUse the table name when checking configured storage columns.
scanStorageColumnsproducesstorageColumns[table] = [column]. This code indexesstorageColumns[key], wherekeyis the document field name. For example, a configuredusers.avatarIdfield checksstorageColumns.avatarId, so it remains an old storage ID.Pass the table name to
remapStorageReferences. CheckstorageColumns[table]?.includes(key)when a mapping exists. Passoptions.tablefor bare-table imports.Proposed fix
const remapStorageReferences = ( document_: Record<string, unknown>, storageIdMap: Map<string, string>, + table: string | undefined, storageColumns?: Record<string, string[]>, ): Record<string, unknown> => { ... - remapped[key] = !storageColumns || storageColumns[key] ? (storageIdMap.get(value) ?? value) : value; + remapped[key] = !storageColumns || storageColumns[table ?? ""]?.includes(key) + ? (storageIdMap.get(value) ?? value) + : value;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer.ts` around lines 695 - 709, Update remapStorageReferences in data-transfer so configured storage columns are checked against the table name, not the document field name. Use storageColumns[table]?.includes(key) when deciding whether to rewrite plain string storage IDs, and pass the current table name from the import path (including options.table for bare-table imports) so scanStorageColumns mappings like storageColumns[table] continue to apply correctly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 657-659: Update migrateStorageBlobs to accept a keyPrefix
parameter and use it instead of the hardcoded empty string; in
packages/cli/src/commands/data-transfer.ts lines 657-659, add the parameter and
apply it, and at lines 852-856 retain mapping?.keyPrefix ?? "" and pass that
value to migrateStorageBlobs.
- Around line 35-45: Update readImportConvexMapping in data-transfer.ts so only
a missing import-convex.json falls back to undefined and the auto-detected
behavior; do not swallow invalid JSON or other read failures in the catch.
Validate the parsed mapping before returning it by checking keyPrefix and every
storageColumns entry, and report/fail schema or parsing issues instead of
logging the optional-file message. Use the existing readImportConvexMapping
symbol and keep the fallback path limited to the missing-file case.
---
Outside diff comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 695-709: Update remapStorageReferences in data-transfer so
configured storage columns are checked against the table name, not the document
field name. Use storageColumns[table]?.includes(key) when deciding whether to
rewrite plain string storage IDs, and pass the current table name from the
import path (including options.table for bare-table imports) so
scanStorageColumns mappings like storageColumns[table] continue to apply
correctly.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 58d20b6d-11b3-4a06-a10e-3b93ba679646
📒 Files selected for processing (1)
packages/cli/src/commands/data-transfer.ts
- Add --verify flag to lunora import - countConvexSourceRows counts source lines per table (excluding _storage) - Post-import parity check: inserted vs source per table, non-zero exit on mismatch - lint:types + ESLint + Prettier clean; 1078 CLI tests pass
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 940-942: Validate the --verify option before the source-row
counting logic in the command flow around countConvexSourceRows: reject
verification when convexTables is undefined, such as plain NDJSON input, with
the command’s established error handling and nonzero result. Preserve the
existing Convex export verification path.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0641cdfc-3ba3-44a2-a837-f0b80e226ff3
📒 Files selected for processing (3)
packages/cli/src/commands/data-transfer.tspackages/cli/src/commands/import/handler.tspackages/cli/src/commands/import/index.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/cli/src/commands/import/handler.ts
- packages/cli/src/commands/import/index.ts
- New convex-snapshot module abstracts directory vs .zip Convex exports - readSnapshotLines/readSnapshotStorageBlob/readSnapshotText resolve entries in both layouts - convexExportTables replaced by resolveConvexSnapshot + listConvexSnapshotTables - Import, _storage migrate, --scan, and --verify all read through the snapshot - adm-zip added to the prod catalog + CLI deps - lint:types + ESLint + Prettier clean; 1078 CLI tests pass
- from-convex.mdx step 5 rewritten for --include-file-storage + --with-storage - Document --scan, lunora/import-convex.json, snapshot.zip, and --verify - import --help examples updated for the storage/scan/verify flags - Mark plan 304 status landed (W1-W5 done)
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/convex-snapshot.ts`:
- Line 29: Update ConvexSnapshot and resolveConvexSnapshot() so ZIP snapshots
create and retain a single AdmZip reader during resolution. Pass that retained
reader through listConvexSnapshotTables, readSnapshotLines,
readSnapshotStorageBlob, and readSnapshotText, removing their separate AdmZip
construction while preserving existing directory-snapshot behavior.
- Around line 140-143: Update readSnapshotStorageBlob for directory snapshots to
resolve both the `_storage` root and the blob path with realpath, then reject
the blob path unless it remains within the resolved storage root before calling
readFile. Preserve the existing blob read behavior for valid in-storage paths
and ensure traversal or symlink escapes are rejected.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 5fad2729-638b-49cc-85cd-98f3373e8f22
⛔ Files ignored due to path filters (3)
plans/304-convex-blob-migration-and-import-verification.mdis excluded by none and included by nonepnpm-lock.yamlis excluded by!**/pnpm-lock.yamland included by nonepnpm-workspace.yamlis excluded by none and included by none
📒 Files selected for processing (5)
apps/docs/src/content/docs/migrating/from-convex.mdxpackages/cli/package.jsonpackages/cli/src/commands/convex-snapshot.tspackages/cli/src/commands/data-transfer.tspackages/cli/src/commands/import/index.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/cli/src/commands/import/index.ts
- packages/cli/src/commands/data-transfer.ts
The flag never reached `runImportCommand`: `ImportOptions` declared
`withstorage`, but cerebro camelCases `with-storage` to `withStorage`, so
`options.withstorage` was always undefined and every `--with-storage` run
silently imported documents only.
Correctness of what it now runs:
- Storage columns were checked against the document's field name rather
than `storageColumns[table]`, so a configured column was never rewritten.
- The remap only visited top-level fields and only ran on lines containing
the literal `_storage`/`$storage`, so nested and plain-string references
survived unrewritten while the import reported success. The walk is now
recursive and every envelope is parsed.
- A rewritten envelope was rebuilt as `{ doc, table }`, dropping any other
field and relabelling a non-string table to `""`. It is rebuilt from the
parsed original and a missing string `table` fails the row.
- The export's `sha256` ships as base16 or base64 depending on version and
was consumed verbatim as the R2 key, the `expectedSha256` parameter, and
the comparand for the worker's lowercase-hex digest. It is normalised
once at the metadata boundary; an unusable `_id`/digest/size fails the
read instead of being skipped.
- A malformed `import-convex.json` fell back to auto-detection, turning a
configured rewrite into a silent no-rewrite. Only a missing file is now
optional; `keyPrefix` is applied instead of being discarded.
- `--with-storage`, `--scan`, and `--verify` against a plain NDJSON source
were ignored; they now exit non-zero. `--scan` also ran after the import
guard could no longer stop it, wrote nothing despite documenting a
mapping file, and needed an admin token to inspect a local export.
Security and durability:
- A `_storage` `_id` was joined onto the snapshot path unchecked, so a
crafted export could read a local file through `../` or a symlink and
upload it to the bucket. Directory reads resolve through realpath and
are rejected outside the snapshot's `_storage` root.
- The signed-URL route cast a caller-supplied `method`, so `?method=DELETE`
minted a signed destructive URL. Validated against a GET/PUT allowlist.
- Post-upload verification demanded a `sha256` the object list only carries
when the writer supplied one, and dereferenced `objects` unguarded. Size
is always compared, the digest when exposed, and an object that fails is
deleted so a later run cannot skip it as already-migrated.
The upload route buffered under the shared 1 MiB JSON cap, which pushed
nearly every photo onto the signed-PUT path; it now declares a 32 MiB
budget the way the KV value PUT does, and the entry-point Content-Length
check reads a per-route table so it agrees with the reader.
`--verify` gains the dangling-storage report it promised, blob uploads skip
what is already present at the same content-hash key, and the blob path is
driven through the injected fetch so it is covered by tests.
304 was marked DONE with every done-criterion unchecked and all five open questions unanswered, so the decisions the implementation had already made lived only in the code. Record them: idempotency by content-hash key with no --force, legacy ids needing no second resolution path, export parity deferred with a reason, an empty keyPrefix default, and the FK spot check cut because preserved ids make it re-derive what row parity proves. Add the 32 MiB body cap the blob path forced, and the review findings the contract now has to hold (flags that cannot apply fail, malformed mappings never degrade to a default, source-supplied ids are paths until checked). 305 planned tables, auth, and storage but never said what "full migration" excludes, so an operator would have discovered their Realtime Database, enum columns, or numeric money column missing at run time. It now carries a per-asset coverage table with an explicit verdict on each row, a lossless reshape rule, a verification workstream matching 304's --verify/--scan, and two new questions (RTDB, storage resumability) instead of silence.
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/cli/src/commands/data-transfer.ts (1)
483-491: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd table and line context to the parse error.
JSON.parse(line)throws a bareSyntaxErrorhere. The operator gets no table name and no line number. The other two parse sites in this file (scanStorageColumns,processLine) both wrap the failure with that context.♻️ Proposed change
async function* wrapJsonlLines(snapshot: ConvexSnapshot, tableEntry: ConvexSnapshotTable): AsyncGenerator<string> { + let lineNumber = 0; + for await (const raw of readSnapshotLines(snapshot, tableEntry)) { const line = raw.trim(); + + lineNumber += 1; if (line.length > 0) { - yield `${JSON.stringify({ doc: JSON.parse(line) as unknown, table: tableEntry.table })}\n`; + let document: unknown; + + try { + document = JSON.parse(line); + } catch (error: unknown) { + throw new LunoraError( + "INTERNAL", + `${tableEntry.table}/documents.jsonl line ${String(lineNumber)}: invalid JSON — ${error instanceof Error ? error.message : String(error)}`, + { cause: error }, + ); + } + + yield `${JSON.stringify({ doc: document, table: tableEntry.table })}\n`; } } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer.ts` around lines 483 - 491, Update wrapJsonlLines to track the current line number while iterating readSnapshotLines, catch JSON.parse failures, and rethrow or wrap the error with both tableEntry.table and the line number, matching the contextual error handling used by scanStorageColumns and processLine.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 938-954: Update the value-remapping flow around remapValue to
track unmigrated storage IDs separately from migrated IDs found in ambiguous
nested positions. Make --verify fail only for IDs absent from storageIdMap, and
report nested ambiguous references with a distinct warning that does not suggest
adding a top-level column or claim the blob was not migrated. Preserve table
context for every occurrence rather than overwriting entries when the same ID
appears in multiple tables.
---
Nitpick comments:
In `@packages/cli/src/commands/data-transfer.ts`:
- Around line 483-491: Update wrapJsonlLines to track the current line number
while iterating readSnapshotLines, catch JSON.parse failures, and rethrow or
wrap the error with both tableEntry.table and the line number, matching the
contextual error handling used by scanStorageColumns and processLine.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 65664bbf-56dc-46f9-9c18-b5497bdc69ed
⛔ Files ignored due to path filters (8)
packages/cli/__tests__/commands/data-transfer.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/import-storage.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/seed.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/storage-admin.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**plans/304-convex-blob-migration-and-import-verification.mdis excluded by none and included by noneplans/305-supabase-firebase-importers.mdis excluded by none and included by nonepnpm-lock.yamlis excluded by!**/pnpm-lock.yamland included by nonepnpm-workspace.yamlis excluded by none and included by none
📒 Files selected for processing (8)
apps/docs/src/content/docs/migrating/from-convex.mdxpackages/cli/package.jsonpackages/cli/src/commands/convex-snapshot.tspackages/cli/src/commands/data-transfer.tspackages/cli/src/commands/import/handler.tspackages/cli/src/commands/import/index.tspackages/runtime/src/create-worker.tspackages/runtime/src/storage-admin-routes.ts
🚧 Files skipped from review as they are similar to previous changes (5)
- packages/cli/src/commands/import/handler.ts
- packages/cli/src/commands/convex-snapshot.ts
- apps/docs/src/content/docs/migrating/from-convex.mdx
- packages/cli/src/commands/import/index.ts
- packages/cli/package.json
The generated `storageSignedUrl` adapter declared only
`{ bucket, expiresInSeconds }`, so the two options the runtime contract and
`@lunora/storage` both accept were dropped on the floor. `getSignedUrl` then
defaulted to `method: "GET"`, and since the method is baked into the HMAC
canonical, a caller asking for a presigned PUT received a URL that verifies
as a GET — every large-blob upload through it fails the signature check.
Nothing caught it because the route's own tests assert against a `vi.fn`
double: the route forwards both options correctly, and the only shipped
producer of that function discarded them.
`lunora import --with-storage` is the first caller to ask for a PUT URL, so
this is the change that makes its above-the-cap path reachable at all.
`--verify` had three ways to reach the wrong answer: - An empty source table failed parity. The endpoint returns no `inserted` entry for a table it received no rows for, and `undefined !== 0` — so a correct import of any export carrying an empty table exited 1 with "inserted 0 of 0 source rows (0 missing)". - A re-run always failed. Rows already present come back as conflicts, which were never credited toward parity, so the second run of the documented resume path reported every table short. - It passed over an unmigrated snapshot. Without `--with-storage` the storage report was skipped entirely, so an export full of file references got a clean bill of health while every one of them still pointed at a Convex id resolving to nothing. That combination is now rejected up front when the export actually carries blobs. The dangling report also conflated two problems under one message and one exit code. A reference whose blob does not exist is broken data no mapping can fix, and fails. A string matching a migrated blob in a column the mapping does not name may equally be user text that happens to look like an id, so it warns and names the column to add. Nested plain strings under a mapped column are now rewritten too — `storageColumns` cannot address a nested path, so the old rule reported references the operator had no way to resolve. With no mapping file at all, every top-level column matching a storage id was silently rewritten, contradicting the CLI's own log line, the guide, and the plan. Rewriting now requires the column to be named, which is what `--scan` is for, and removes the inversion where adding a mapping made rewriting less aggressive. Other corrections: a failed batch discarded every accumulated tally instead of reporting how far the run got; 500-row batches could exceed the endpoint's 1 MiB body cap, so batching now tracks bytes as well as rows; the object listing paged 100 at a time over the whole bucket; a failed delete-on-mismatch was reported as a successful removal; non-`_storage` Convex system tables were streamed in as application data; and a mistyped path reported "not a Convex export" rather than "no such file". Blob uploads now run eight at a time. They carry independent content-hash keys and independent verification, so nothing ordered them — serially, a 50k-blob migration is round-trip-bound for hours. Structurally, `data-transfer.ts` had grown to 1,652 lines holding seven concerns. It is now a directory of focused modules, matching how every other multi-file command in this package is laid out, and `runImportCommand` is a sequence of named phases rather than a 313-line function. Every `sonarjs/cognitive-complexity` suppression in the touched files is gone — none of them were earned.
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (4)
packages/cli/src/commands/data-transfer/export.ts (1)
71-117: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueFlush the decoder after the last chunk.
decoder.decode(read.value, { stream: true })keeps a partial multi-byte sequence buffered. If the response ends inside such a sequence, those bytes never reach the sink. Calldecoder.decode()once after the read loop and append the result toleftoverbefore the final write.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer/export.ts` around lines 71 - 117, Update streamNdjsonToSink to flush the TextDecoder after the read loop by calling decoder.decode() without a chunk and appending the result to leftover before the final leftover write, preserving the existing row counting and sink backpressure behavior.packages/cli/src/commands/data-transfer/storage-blobs.ts (1)
134-161: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueGuard the cursor loop against a repeated cursor.
The
do ... whileloop followsjson.cursorwhiletruncatedis true. If a host returns the same cursor twice, the loop never ends andobjectsgrows without bound. Track the previous cursor and stop when it repeats.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer/storage-blobs.ts` around lines 134 - 161, The listStorageObjects cursor loop must detect repeated pagination cursors. Track the previously used cursor and only continue when truncated is true, a cursor is present, and it differs from the prior cursor; otherwise stop iteration while preserving the existing object accumulation behavior.packages/cli/src/commands/data-transfer/import-rows.ts (1)
50-56: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueGuard the envelope parse like the bare-document parse.
wrapBareDocumentconverts aSyntaxErrorinto a line-scopedLunoraError.remapEnvelopeparses without a guard, so a malformed line raises a rawSyntaxErrorwith no line number. Today the input on this path is generated bywrapJsonlLines, so it is always valid JSON. Adding the same guard keeps the two paths consistent if the source ever changes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer/import-rows.ts` around lines 50 - 56, The remapEnvelope function must handle JSON.parse failures consistently with wrapBareDocument. Wrap the parse of trimmed in a SyntaxError guard and rethrow a line-scoped LunoraError that includes lineNumber, while preserving the existing validation for a missing or non-string table.packages/cli/src/commands/convex-snapshot.ts (1)
154-173: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueUse async ZIP blob reads under the upload concurrency window.
Promise.allSettled(pending.slice(...) map uploadOne)starts several blob reads at once, andsnapshot.zip.readFile(...)decompresses each entry synchronously. For large ZIP snapshots, make these reads usereadFileAsyncso the concurrent uploads can overlap instead of blocking the event loop during each blob read.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/convex-snapshot.ts` around lines 154 - 173, Update readSnapshotStorageBlob to use the ZIP reader’s asynchronous read method, readFileAsync, for non-directory snapshots instead of snapshot.zip.readFile; preserve the existing missing-blob validation and Buffer conversion, while leaving directory-backed reads unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/data-transfer/export.ts`:
- Around line 59-65: Update writeWithBackpressure to listen for both drain and
error while awaiting backpressure, resolving on drain and rejecting with the
stream error so the caller’s catch path invokes discardPartialExport. Also
ensure the file write stream has an error listener or otherwise routes write
failures through the same rejection path, preventing unhandled error events.
In `@packages/cli/src/commands/data-transfer/import-batcher.ts`:
- Around line 113-123: Update the push function to check whether adding the
incoming row would exceed config.maxBatchBytes or config.batchSize before
appending it; flush the existing batch first when needed, then append the row
and update batchBytes. Preserve handling for rows that individually exceed the
byte limit so they can still be sent without creating an oversized combined
batch.
In `@packages/cli/src/commands/data-transfer/storage-blobs.ts`:
- Around line 256-266: Update the sizeMismatch check in the post-upload
verification flow to compare stored.size with metadata.size only when
stored.size is present, treating an omitted size as unverifiable rather than
mismatched. Emit a warning when size is absent, while preserving the existing
hash validation and deletion behavior for confirmed mismatches.
In `@packages/cli/src/commands/data-transfer/storage-mapping.ts`:
- Around line 129-143: Update the documentation comment above the recursive
storage-value walk to remove the claim that plain strings are rewritten when no
mapping file exists. Keep the description that plain strings are rewritten only
under columns listed by isMappedColumn, matching the existing behavior and the
documentation near isMappedColumn.
- Around line 226-232: Update holdsStorageId to recursively inspect nested plain
objects and arrays, matching the traversal behavior of remapValue so any string
value at any depth is checked against storageIds. Preserve the existing
direct-string and array handling while adding recursive object traversal and
returning true as soon as a matching storage ID is found.
---
Nitpick comments:
In `@packages/cli/src/commands/convex-snapshot.ts`:
- Around line 154-173: Update readSnapshotStorageBlob to use the ZIP reader’s
asynchronous read method, readFileAsync, for non-directory snapshots instead of
snapshot.zip.readFile; preserve the existing missing-blob validation and Buffer
conversion, while leaving directory-backed reads unchanged.
In `@packages/cli/src/commands/data-transfer/export.ts`:
- Around line 71-117: Update streamNdjsonToSink to flush the TextDecoder after
the read loop by calling decoder.decode() without a chunk and appending the
result to leftover before the final leftover write, preserving the existing row
counting and sink backpressure behavior.
In `@packages/cli/src/commands/data-transfer/import-rows.ts`:
- Around line 50-56: The remapEnvelope function must handle JSON.parse failures
consistently with wrapBareDocument. Wrap the parse of trimmed in a SyntaxError
guard and rethrow a line-scoped LunoraError that includes lineNumber, while
preserving the existing validation for a missing or non-string table.
In `@packages/cli/src/commands/data-transfer/storage-blobs.ts`:
- Around line 134-161: The listStorageObjects cursor loop must detect repeated
pagination cursors. Track the previously used cursor and only continue when
truncated is true, a cursor is present, and it differs from the prior cursor;
otherwise stop iteration while preserving the existing object accumulation
behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 05a811de-b04d-4ef8-8f26-82691827ebc5
⛔ Files ignored due to path filters (6)
examples/blog/lunora/_generated/app.tsis excluded by!**/_generated/**and included by noneexamples/team-chat/lunora/_generated/app.tsis excluded by!**/_generated/**and included by nonepackages/cli/__tests__/commands/import-storage.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/migrate.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/codegen/__tests__/fixtures/simple/expected/_generated/app.tsis excluded by!**/_generated/**,!**/__tests__/**and included bypackages/**pnpm-workspace.yamlis excluded by none and included by none
📒 Files selected for processing (14)
apps/docs/src/content/docs/migrating/from-convex.mdxpackages/cli/src/commands/convex-snapshot.tspackages/cli/src/commands/data-transfer.tspackages/cli/src/commands/data-transfer/export.tspackages/cli/src/commands/data-transfer/import-batcher.tspackages/cli/src/commands/data-transfer/import-rows.tspackages/cli/src/commands/data-transfer/import-source.tspackages/cli/src/commands/data-transfer/import-verify.tspackages/cli/src/commands/data-transfer/import.tspackages/cli/src/commands/data-transfer/index.tspackages/cli/src/commands/data-transfer/shared.tspackages/cli/src/commands/data-transfer/storage-blobs.tspackages/cli/src/commands/data-transfer/storage-mapping.tspackages/codegen/src/emit-app.ts
💤 Files with no reviewable changes (1)
- packages/cli/src/commands/data-transfer.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- apps/docs/src/content/docs/migrating/from-convex.mdx
Four defects from review of the split, three of them mine:
- `push` appended the row and flushed afterwards, so a POSTed body could
exceed `maxBatchBytes` by one whole row. With the 900 KB ceiling against
the endpoint's 1 MiB cap, a single row over ~148 KiB produced the 413 the
ceiling exists to prevent. The byte check now runs before the row joins the
batch; a row wider than the whole budget still goes on its own, which is
the best that can be done for one document.
- Post-upload verification treated an absent `size` in the object listing as
a mismatch and DELETED the blob it had just uploaded correctly. `sha256`
was already tolerant of absence; both fields are optional, and missing
means "not reported", not "does not match".
- `--scan` inspected only top-level strings and top-level arrays, while the
rewrite descends to any depth under a mapped column. A column holding
`{ file: "<storageId>" }` was therefore never proposed, leaving the
operator with a reference reported as ambiguous and no column to add for
it. The scan walk now matches the rewrite walk, and skips the
self-describing `{ $storage }` form, which needs no mapping entry.
- `remapStorageReferences`' docblock still claimed plain strings are
rewritten "with no mapping file at all" — the behaviour the previous
commit deliberately removed.
Also in `export`, pre-existing but now load-bearing: `writeWithBackpressure`
waited only for `drain`, so a sink failing mid-wait (full disk, EPIPE on a
closed pipe) emitted `error`, never `drain`, and hung the export forever. The
file stream also had no `error` listener, making any write failure an
unhandled event that takes the process down instead of failing the command.
_storage blobs to R2
`api:check` runs as its own CI job and sees changes the linters cannot, so a green lint/test locally says nothing about it. Both entries are intentional and already reviewed in this branch: `ImportCommandOptions` gains `scan`, `verify`, and `withStorage` for the new flags, and `StreamingFetchLike` widens its body to `string | Uint8Array` so the blob upload path can send bytes through the same injected fetch the rest of the command uses. Regenerated with `pnpm run api:update` after a fresh `build:packages` — it reads `dist/`, so a stale build writes a wrong snapshot.
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Summary
lunora importmoved a Convex export's documents but not its files. The_storagetable was skipped with a warning telling the operator to "upload theexported blobs to R2 and re-point the keys" by hand — so a Convex app with file
storage lost every file, and every field holding a storage id became a dangling
reference. This closes that gap and adds the verification the command never had.
--with-storageuploads every blob under a content-hash key(
sha256hex of the bytes) through a checksum-verified admin route, thenrewrites the documents' references to the new keys. Blobs land before documents,
so no imported row can reference an object that is not there yet. Re-runs are
cheap and safe: one prefix listing tells the CLI what is already present at the
right size, and those are mapped without re-uploading.
Reference rewriting distinguishes what it can know from what it cannot.
{ $storage: id }is Convex's self-describing form, so it is rewritten whereverit occurs at any depth. A plain string is ambiguous against ordinary text, so it
is rewritten only under a column
lunora/import-convex.jsonnames — which iswhat
--scangenerates, by exact-matching source values against the_storageid set. It writes the file withwx, so a reviewed mapping is neverclobbered by a re-scan, and it imports nothing.
--verifychecks that every source row is accounted for and that everystorage reference resolved. The two failure modes are kept apart because they
have different remedies: a reference whose blob does not exist is broken data no
mapping can fix, and fails the run; a string matching a migrated blob in an
unnamed column may equally be user text that happens to look like an id, so it
warns and names the column to add.
Directory exports and
snapshot.zipare both accepted.Also in here
@lunora/runtime) —PUT /_lunora/admin/storageacceptsexpectedSha256/expectedSize, digests the body, and rejects withSTORAGE_CHECKSUM_MISMATCHbefore anything is written. The route declaresa 32 MiB body budget of its own the way the KV value PUT already does; without
it the shared 1 MiB JSON cap would push nearly every photo onto the fallback
path.
/_lunora/admin/storage/urlnow validatesmethodagainst a GET/PUTallowlist rather than casting it — an unchecked
?method=DELETEwould haveminted a signed destructive URL.
@lunora/codegen) — the generatedstorageSignedUrladapterdeclared only
{ bucket, expiresInSeconds }and dropped themethodandcontentTypethat both the runtime contract and@lunora/storageaccept.getSignedUrlthen defaulted toGET, and since the method is baked into theHMAC canonical, any caller asking for a presigned PUT received a URL that
verifies as a GET. This is the first caller to ask for one.
data-transfer.tshad grown to ~1,650 lines holding sevenconcerns. It is now a directory of focused modules, matching how every other
multi-file command in
@lunora/cliis laid out, andrunImportCommandis asequence of named phases rather than one long function. Every
sonarjs/cognitive-complexitysuppression in the touched files is gone — theywere removed by extraction, not by widening the threshold.
plans/304records the answers to its open questions (content-hashkeys, skip-if-present idempotency without a
--force, the cut FK spot checkand why, the 32 MiB cap).
plans/305gains a per-asset coverage table so theSupabase/Firebase work states up front what "migrated" does and does not
include.
Known limitation
Objects above 32 MiB cannot reach the worker, so they fall back to a signed
PUT. That path needs the app to have signed URLs configured and to servethe route those URLs address — Lunora does not mount one — and it is verified
only after the write (by size, plus SHA-256 where the bucket records one, which
R2's
list()does not). An object that fails verification is deleted rather thanleft to look already-migrated.
from-convex.mdxsays all of this, and points atwrangler r2 object putfor anyone without a signed-PUT route.Linked issues
plans/304-convex-blob-migration-and-import-verification.md(W1–W5)plans/305-supabase-firebase-importers.md(its storage transferreuses the verified upload route)
Test plan
pnpm --filter "@lunora/cli" run test— 1111 passing (33 new, coveringblob upload, both reference shapes, nested rewriting, base64 vs hex
digests, idempotent re-runs,
keyPrefix, ZIP snapshots, the signed-PUTpath including delete-on-mismatch, cursor paging, byte-ceiling batching,
partial-failure reporting, and every flag guard)
pnpm --filter "@lunora/runtime" run test— 844 passing (checksummismatch rejects before write, an upload above the shared JSON cap
succeeds, the signed-URL method allowlist)
pnpm --filter "@lunora/codegen" run test— 1054 passing, golden fixturesand example
_generated/app.tsregeneratedpnpm --filter "…" run lint:typesclean for cli / runtime / codegeneslintandprettier --checkclean on every touched filepnpm run lint:package-json— 74/74 sorteddist/smoke-tested end-to-end through the ZIP path, to confirmadm-zip's CJS interop survives bundling into the ESM-only CLIChecklist
apps/docs(migrating/from-convex.mdx)