Repository navigation
feat(import): full Supabase and Firebase migration — rows, auth, and files - #357
Conversation
`lunora import --from supabase|firebase` turns a foreign dump into the same
`{ table, doc }` NDJSON the Convex importer already feeds, so ids preserve
verbatim and foreign keys survive without a second remapping pass.
Supabase reads a directory of `COPY … TO STDOUT WITH CSV HEADER` files through
`csv-parse` — quoting, embedded newlines, and doubled quotes are exactly what a
hand-rolled split gets wrong on the one file an operator cannot re-create. NULL
is distinguished from the empty string the way Postgres CSV encodes it: an
unquoted empty field is NULL, a quoted `""` is a real empty string. Collapsing
the two would null every empty text column in the database.
Firebase decodes the Firestore typed-value JSON — `{ integerValue: "42" }`,
`{ timestampValue }`, `{ mapValue: { fields } }`, references, geopoints, bytes —
from a directory of per-collection files, in any of the three container shapes
those tools emit.
Reshapes are declared per column in `lunora/import-<source>.json`, from a closed
set with defined lossless targets. The rule the reshape module exists to enforce
is that a conversion which would lose information errors and names the column:
an `int8` past `Number.MAX_SAFE_INTEGER` or a `numeric` with more digits than a
double holds refuses rather than rounding, and points at `int8-string`. A column
the mapping does not name is copied through untouched. An unknown reshape name
fails the run instead of being ignored, and a malformed mapping never degrades
to "no mapping" — dropping it would turn a declared conversion into a silent
pass-through, which is the corruption the rule exists to prevent.
Firestore's `integerValue` is decoded the same way: it arrives as a string
precisely because it may exceed the safe range, and it stays one when it would
not survive the conversion.
Note on the plan: 305's D3 describes `gcloud firestore export` as emitting "JSON
document shards". It does not — that command writes LevelDB-log files wrapping
protobuf entities. What it describes is the Firestore REST/RPC JSON encoding,
which is real, is what every JSON-producing Firestore read path emits, and is
the part worth owning. The decoder targets that; the guide will document how to
produce it.
`--with-storage` now moves a foreign bucket into R2 alongside the rows, and rewrites the columns a mapping names from the provider-side path to the R2 key. Supabase objects are listed and downloaded live through its Storage REST API. The service-role key is read from `SUPABASE_SERVICE_ROLE_KEY` rather than taken as a flag: it grants full read/write on the project, and a flag puts it in the process table where any other local process can read it — the same reason `--token` already carries that warning. Firebase has no listing endpoint the CLI can reach without owning Google's auth, so its bucket arrives as a directory the operator downloaded with `gcloud storage cp -r` and passes as `--storage-dir`. Both then take the identical path into R2 — content-hash key, checksum-verified upload, skip-if-present, delete-on-mismatch — inherited from the Convex blob migration rather than rebuilt. A bucket migration is the one part of an import genuinely expensive to repeat: every object is a download from the old provider and an upload to the new one, so a run that dies at object 40,000 of 50,000 has spent real time and bandwidth. Each completed transfer is appended to a checkpoint, and a re-run skips what is already recorded — without re-downloading it. The checkpoint is append-only NDJSON rather than a rewritten blob: appending costs no re-serialisation as the run grows, and a process killed mid-write leaves at most a torn final line, which is skipped at the cost of one repeated object rather than a corrupt file that strands the whole migration. Progress is reported as it goes, so a multi-hour transfer is visibly moving. A partial transfer does not then import rows. Every path column would point at an object that is not there yet, which is the dangling reference the files-first ordering exists to prevent — so the run stops, says the checkpoint is saved, and exits non-zero. A path the transfer never saw is reported and left as-is rather than guessed at, and fails `--verify`. Deviates from the plan's D5, which specified `@aws-sdk/client-s3`: Supabase's own REST API takes the service-role key the operator already has, so this needs no S3 client at all. That removes the plan's own listed bundle-size risk and one credential the operator would otherwise have to create.
A `lunora/import-<source>.json` `auth` block turns a Supabase `auth.users` dump (plus `auth.identities` for the linked providers) or a Firebase `auth:export` JSON into better-auth `user` and `account` rows. Users are emitted before accounts, so an account row never lands before the user it points at. Passwords are never carried across. Supabase stores bcrypt, Firebase its own scrypt variant, and better-auth hashes with neither — there is no honest conversion, and inventing one locks every user out with an error they cannot act on. Profile fields and `emailVerified` do carry, so a reset is the only thing a user redoes. Enforcing that turned out to need more than not reading the hash column: the auth dumps were also being picked up as ordinary tables, so `auth.users.csv` imported as a table named `auth.users` with `encrypted_password` intact. A test asserting the bcrypt prefix never appears in the output caught it. Files a mapping claims for auth are now excluded from the table listing. Two users claiming one email fails the run with both ids named, rather than silently merging two people into one identity. `--scan` now covers the new sources. For Supabase it samples each column and proposes a reshape, requiring every non-null value to agree — one stray value falls back to "copy it through untouched", which is the safe answer, since an un-reshaped column keeps its text while a wrong reshape either errors or changes the data. `int8-string` is only proposed when a value genuinely exceeds the safe range, so ordinary id columns are left alone. Firestore values are already self-describing, so its skeleton just lists the collections with the `storageColumns` an operator still has to declare. Neither overwrites a mapping that already exists.
`migration_stale_import` catches the half-finished port: the data moved, most handlers were rewritten, and one still imports `@supabase/supabase-js` or `firebase/firestore`. Nothing fails — it compiles and typechecks — so the app quietly keeps reading from the platform it just left and the two stores drift apart until someone notices stale data. The finding names the guide that covers finishing the port. Only the three platforms with migration guides are matched: the lint exists to catch an unfinished migration, not to police library choice, so a deliberate second data source just has to live outside `lunora/`. Evidence comes from a codegen feeder that reads real import declarations rather than grepping — a specifier is only a specifier when the AST says so, and a mention in a comment or a string is not an import. Prefix matching is boundary-aware, so `convex` and `convex/server` match while `convexity` does not.
Both guides told the reader to dump each table, reshape it by hand, and batch-insert through a mutation or a `lunora run` script — a homework assignment rather than a migration path. They now walk the importer end to end: dumping, generating the column mapping with `--scan`, importing with `--verify`, the auth import, and the storage transfer. Each guide closes with what the importer does NOT move — RLS policies, edge and cloud functions, triggers, realtime listeners, sessions — linked back to the step that covers it, so a reader who followed the import knows exactly what is still theirs to do. Firebase says plainly that Realtime Database is a different format with no reader, rather than leaving it to be discovered mid-cutover. Two things the guides now state that the previous versions could not: - `gcloud firestore export` is the wrong artifact — it writes protobuf, not JSON. The guide gives the Admin SDK script that produces what the reader wants. - Passwords cannot migrate from either platform, so every user signs in once via "forgot password". The announcement needs writing before cutover, not after. Also adds three engineering posts on the migration work: why preserving ids makes a Convex import single-pass, why the Supabase reader refuses a conversion that would drop a digit, and what Firestore's typed-value encoding is actually for. Plan 305 records both design deviations (the Firestore format, and dropping the S3 SDK for Supabase's own REST API) and answers its eight open questions.
Six defects, five of which ended in a green `--verify` line over data that was
already wrong. Every one is now covered by a regression test.
**Credential material reached the wire.** The `auth.*` dump exclusion was built
only from what the mapping file named, but the mapping is optional — so a dump
imported without one sent `auth.users.csv` as an ordinary table, carrying the
bcrypt hash AND the live single-use `recovery_token`, `confirmation_token` and
`email_change_token_*` values. Exit 0. The docs and the code comment both
asserted the opposite guarantee. Exclusion is now by shape (`auth.` prefix) with
the mapping as an addition, and any row carrying a known credential column is
refused outright rather than filtered — a silent filter would hide that the file
is an auth dump.
**Rows silently lost their id.** `toDocument` set `_id` only on finding the id
column, with no check that it ever appeared. A table whose PK is `user_id`
imported with no `_id` at all, the target minted fresh ones, and every foreign
key dangled — with `--verify` green, because the row counts still matched. That
defeats the premise the whole importer rests on, so it is now a hard error
naming the table and the columns present.
**The transfer checkpoint was trusted without checking the target.** It records
source paths and nothing about which deployment it was written against, so a
resume against a different or wiped bucket skipped every upload and rewrote
documents to keys that do not exist. The listing is already in hand; a
checkpoint entry now has to appear in it.
**`--verify` passed over unmigrated storage in three ways**: for Convex when the
export was taken without `--include-file-storage` (no `_storage` table, so the
existing guard never fired and `{ $storage }` objects imported verbatim); for
the foreign sources, which had no equivalent guard at all; and for a plain
string in a column the mapping explicitly declared, which was passed through and
reported nowhere — the self-describing form had that check, the declared column
did not.
Also: the run summary printed `logger.success` on a run exiting 1, so the last
line an operator read disagreed with the exit code.
R2 reports a checksum only for objects whose writer supplied one, and `upload()` never did. So `list()`/`head()` always returned no `sha256`, and every downstream integrity check silently degraded to comparing sizes — while the CLI's tests, whose fake bucket does return one, exercised a hash branch that was dead in production. The admin upload route already computes that digest to verify the body before writing, then discarded it. It now travels through to `bucket.put`, which makes R2 both verify the write itself and record the checksum. Two checks become real as a result: the >32 MiB signed-PUT verification, and the idempotency probe, which compared sizes alone and can now compare content. `StorageObject` gains the `sha256` field the CLI was already reading. It existed at runtime but not in the declared contract, so "the host does not report this" was indistinguishable from "the contract never promised it" — the failure mode the platform-parity note calls out after `ShardSqlExec`. Also from review: - `--scan` was a second implementation of the rewrite walk, kept in step with it by a comment. It is now a dry run of the real thing: an identity map and no `storageColumns` makes every plain string matching a storage id land in `ambiguous`, which is exactly the candidate list. Drift is now impossible rather than merely discouraged. - 8 concurrent uploads × a 32 MiB per-blob ceiling is ~256 MiB of request bodies against a 128 MB isolate shared by every concurrent request. The window now budgets bytes as well as requests, so small blobs still run eight wide and large ones narrow it automatically. - `contentType` was the one `_storage` field reaching an HTTP header unvalidated, so a CR/LF crashed the client with an opaque "Invalid header value" instead of naming the offending metadata line. - A list response claiming more pages without advancing its cursor would have looped forever. - `SUPABASE_URL` is refused unless https — the service-role key travels as a Bearer on every request, and over plain HTTP that is the whole credential in cleartext. - A Firestore document with no fields crashed with "Cannot use 'in' operator"; `documents.list` omits `fields` entirely for one. - `timestamp-iso` truncated to milliseconds while Postgres defaults to microseconds, despite having a lossless target available.
Three structural findings, each deleting a copy rather than moving one. **The two foreign readers were parallel implementations.** Matching mapping-named files, excluding the auth dump, erroring on an empty directory, sorting for determinism, seeding the parity tally, emitting auth rows first — all written twice. That duplication had already drifted once: the auth CSV reader lost the `\N` NULL handling, so a `COPY … WITH (FORMAT text)` dump gave every other table proper nulls and the auth table the literal string "\N". `dump-directory.ts` now owns the shape; each source contributes only its decoder, which is the only part that genuinely differs. **`auth-reader.ts` was one seam too many.** It imported five of the six symbols `auth.ts` exported and nothing else did, and `AuthRows` was exported solely to cross that boundary — the header comment defending the split was the tell. Merged. **`import-source.ts` and `import.ts` imported each other.** Type-only, so it erased at build, but it coupled source resolution to the whole command flag surface. Source resolution now declares the handful of flags it actually reads. `CONVEX_STORAGE_TABLE` and `isConvexSystemTable` moved to `convex-snapshot.ts`, which already owns the export layout, instead of being defined in the resolver and imported back out by two other modules.
The live Supabase Storage path was the only third-party integration in the branch and had zero tests: bucket enumeration, offset paging, the folder-placeholder recursion that is the only reason nested objects move at all, and the https guard. It now has them, including an assertion that the service-role key never appears in a URL — it travels as a header, and a test is the only thing keeping it that way. Also covered: the storage-dir symlink guard (a security check with no test), and the `conflicts > 0` parity branch — which exists specifically so a re-run does not fail `--verify`, on the run the docs tell operators to make, and which every fake worker returning `conflicts: 0` had left unexercised. The batcher tests moved out of the storage file. Neither had any storage in them, and both files now describe what they contain. Two smaller review items: `STORAGE_CHECKSUM_MISMATCH` was inserted out of alphabetical order in the error catalog, and `migration_stale_import` — a WARN — sat above the schema-correctness errors that `STATIC_LINTS` documents as coming first. `AdvisorStaleMigrationImport` is now re-exported like every sibling feeder type; the snapshot referenced it without declaring it.
The allowlist guard inlined in `upload()` pushed it past the cognitive complexity ceiling once the sha256 pass-through landed. Lifting it to a named function keeps `upload()` a readable sequence of checks and gives the security control its own documented unit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The directory import reads a table's `documents.jsonl` line by line; the ZIP import inflated the whole entry into a string and then split it into an array of every line. A snapshot's packaging therefore decided whether the import fit in memory — the same export succeeded as a directory and ran the CLI out of heap as an archive. Read the entry's byte range through `zlib` instead, using the local file header for the data offset (its name/extra lengths may differ from the central copy). `readSnapshotText` had one caller left, which now reads lines too, so it is gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A transfer keys objects by their bucket-qualified path, but an app column holds whichever spelling it stored: bucket-qualified, bucket-relative, or a getPublicUrl()/createSignedUrl() URL — the last being what most Supabase apps persist. Only the first matched, so the other two were reported as untransferred with no mapping knob able to fix them. Index the transfer by all three. A bucket-relative name carried by more than one bucket is deliberately left out: resolving it to whichever bucket listed first would rewrite rows at the wrong object, which is worse than reporting it unresolved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- `ImportCommandResult.body` was `unknown`, so a caller wanting the conflict count cast its way there. It is now `ImportSummary`, and the `--scan` path (whose product is the file it writes) returns none. - The two mapping files repeated their validation, phrasing the same mistake two ways; the three checks now live in one place. The shapes stay separate — a Convex export is self-describing, so its mapping has no use for a file name, an id column, or a declared reshape. - The storage-reference walk moves out of the mapping-file module. It has two callers that must not diverge: the rewrite and --scan, which runs it as a dry run. - The foreign path rewrite is a hook on the row transformer rather than a second transform wrapped around it, so a row is parsed and serialised once instead of twice. - `TransferredObject.source` meant the object path while the sibling `source` argument meant the provider; they are now `path` and `provider`. The progress reporter is a logging cadence, not checkpoint state, so it sits with its caller. - The three worker fakes in the source tests shared a copied import handler; now one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughThe PR adds Firebase and Supabase import support, extends Convex storage migration, adds migration documentation, detects stale platform imports, and propagates storage checksums. ChangesMigration import platform support
Advisor and storage contracts
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant SourceReader
participant StorageTransfer
participant R2
participant RowImporter
CLI->>SourceReader: resolve and stream source rows
CLI->>StorageTransfer: transfer source storage
StorageTransfer->>R2: upload or resume objects
StorageTransfer-->>CLI: return path mappings
CLI->>RowImporter: remap and import rows
RowImporter-->>CLI: return typed import summary
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
|
Thank you for following the naming conventions! 🙏 |
There was a problem hiding this comment.
Actionable comments posted: 16
🧹 Nitpick comments (10)
packages/cli/src/commands/data-transfer/sources/reshape.ts (1)
52-53: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMerge the two identical regexes.
EXPONENT_REandEXPONENT_SPLIT_REare both/e/i. One constant covers both uses. Keep one name and delete the other.Note: a
/e/iliteral has nogflag, so sharing it betweentestandsplitcarries nolastIndexrisk.♻️ Proposed simplification
-const EXPONENT_RE = /e/i; -const EXPONENT_SPLIT_RE = /e/i; +const EXPONENT_RE = /e/i;Then use
EXPONENT_REinexpandExponent:- const [mantissa = "0", exponentPart = "0"] = raw.split(EXPONENT_SPLIT_RE); + const [mantissa = "0", exponentPart = "0"] = raw.split(EXPONENT_RE);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer/sources/reshape.ts` around lines 52 - 53, Merge EXPONENT_RE and EXPONENT_SPLIT_RE into a single /e/i constant, remove the duplicate declaration, and update expandExponent and any other references to use the retained EXPONENT_RE symbol.packages/runtime/src/storage-admin-routes.ts (1)
177-180: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueThe checksum is recorded only for uploads that request verification.
sha256staysundefinedunless the caller sendsexpectedSha256orexpectedSize. Every other upload still lands without a recorded checksum, so the degradation the comment describes remains for those objects.If you want every object to carry a checksum, compute the digest unconditionally. The cost is one SHA-256 over a body that
STORAGE_UPLOAD_MAX_BODY_BYTESalready bounds. If the current scope is intentional, narrow the comment to say the digest is recorded only when verification is requested.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/runtime/src/storage-admin-routes.ts` around lines 177 - 180, Update the upload flow around storageUpload so the SHA-256 digest is computed unconditionally for every request, rather than only when expectedSha256 or expectedSize is provided, and pass that digest to storageUpload for recording. If verification-only behavior is intentional instead, narrow the nearby comment to state that the checksum is recorded only for uploads requesting verification.packages/cli/src/commands/data-transfer/sources/storage-progress.ts (1)
85-91: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value
recordTransferrunsmkdirand opens a file handle for every object.Each completed object costs one
mkdirand one open/write/close cycle. For a 50,000-object bucket that is 100,000 extra syscalls. Durability does not require it: an appended write to a handle opened with"a"gives the same one-line-per-object guarantee.Open the checkpoint once per run and reuse the handle, or create the directory only when the file is first written.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer/sources/storage-progress.ts` around lines 85 - 91, Refactor recordTransfer so the checkpoint directory is created and the progress file is opened once per transfer run, then reuse the append-capable file handle for each completed object. Preserve one JSON line per entry and ensure the handle is closed after the run, rather than performing mkdir and open/write/close operations for every object.packages/storage/src/create-storage.ts (2)
323-331: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueValidate the
sha256format before handing it to R2.
uploadOptions.sha256is forwarded unchecked. R2 requires a 64-character hex string or a 32-byte buffer. A truncated or non-hex value produces an opaque binding error instead of theVALIDATION_ERRORthe rest of this module raises for bad input.Add a short guard next to
assertContentTypeAllowedso a malformed digest names the field.♻️ Proposed guard
+const SHA256_HEX_RE = /^[\da-f]{64}$/i; + +/** Reject a digest R2 would refuse, so the error names the option rather than the binding. */ +const assertSha256Valid = (sha256: ArrayBuffer | string | undefined): void => { + if (sha256 === undefined) { + return; + } + + if (sha256 instanceof ArrayBuffer) { + if (sha256.byteLength !== 32) { + throw new LunoraError("VALIDATION_ERROR", "`@lunora/storage`: sha256 buffer must be exactly 32 bytes"); + } + + return; + } + + if (!SHA256_HEX_RE.test(sha256)) { + throw new LunoraError("VALIDATION_ERROR", "`@lunora/storage`: sha256 must be 64 hex characters"); + } +};🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/storage/src/create-storage.ts` around lines 323 - 331, Validate uploadOptions.sha256 near assertContentTypeAllowed before calling options.bucket.put: accept only a 64-character hexadecimal string or a 32-byte buffer, and otherwise raise the module’s standard VALIDATION_ERROR while naming sha256. Keep valid digests unchanged when constructing the put options.
249-269: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument the
allowedContentTypes+ missingcontentTyperuntime cost.
allowedContentTypeswithoutcontentTypenow rejects uploads withVALIDATION_ERROR; update the Storage docs package example to passcontentTypeand add a migration note.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/storage/src/create-storage.ts` around lines 249 - 269, Update the Storage documentation package example to provide a valid contentType whenever allowedContentTypes is configured, matching assertContentTypeAllowed. Add a migration note explaining that uploads omitting contentType now fail at runtime with VALIDATION_ERROR and must be updated to declare an allowed content type.packages/cli/src/commands/data-transfer/sources/dump-directory.ts (1)
48-52: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPreserve the underlying
readdirfailure.
catch(() => undefined)collapsesENOENT,ENOTDIR, andEACCESinto one message. Attach the original error ascauseso the operator can tell "wrong path" from "no permission".♻️ Proposed refactor
- const entries = await readDirectory(directory).catch(() => undefined); - - if (entries === undefined) { - throw new LunoraError("INTERNAL", `${directory} is not a readable directory`); - } + const entries = await readDirectory(directory).catch((error: unknown) => { + throw new LunoraError("INTERNAL", `${directory} is not a readable directory`, { cause: error }); + });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer/sources/dump-directory.ts` around lines 48 - 52, Update the directory-reading error handling around readDirectory so the caught failure is retained and passed as the cause when constructing LunoraError. Preserve the existing unreadable-directory message and undefined-check behavior while allowing callers to distinguish failures such as ENOENT, ENOTDIR, and EACCES.packages/cli/src/commands/data-transfer/import.ts (2)
104-127: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winReport foreign transfers in
ImportSummary.storage.
buildImportBodyaddsstorageonly whenstorageIdMapis defined.storageIdMapstays undefined for Supabase and Firebase sources, because those sources producetransferredPathsinstead. A foreign import with--with-storagetherefore returns a summary with no storage section, and a caller cannot see how many objects moved.ImportSummaryis exported frompackages/cli/src/commands/data-transfer/index.ts, so this is part of the public type surface.Consider passing the transferred-path count into
buildImportBodyand emittingstoragefor foreign sources too.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer/import.ts` around lines 104 - 127, Update buildImportBody and its callers to accept the transferred-path count, and emit ImportSummary.storage for foreign Supabase and Firebase imports when --with-storage is used, even when storageIdMap is undefined. Populate the storage summary with the number of transferred objects while preserving the existing storage remap fields for sources that provide storageIdMap, and keep the exported ImportSummary type consistent with both paths.
252-272: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueType the
runScanreturn value.
runScanreturnsPromise<unknown>. The caller only tests it againstundefined, so the loose type is not a defect. A union of the scanner return types documents the contract and keeps the--scanexit-code check honest if a scanner later returns a falsy value.As per path instructions,
packages/**/src/**/*.tsrequires "Verify TypeScript types are correct".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer/import.ts` around lines 252 - 272, Update the runScan return type from Promise<unknown> to a union of the concrete return types from scanStorageColumns, scanFirebaseDump, and scanSupabaseDump, plus undefined for the default branch. Preserve the existing scanner dispatch and caller’s undefined check.Source: Path instructions
packages/cli/src/commands/data-transfer/import-verify.ts (1)
135-145: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winLog an error line when
verifyfails on untransferred paths.
reportStorageOutcomelogslogger.errorbefore it returnstrue.reportUntransferredPathsreturnstruewith only warnings logged. The run then exits 1 while no error line names this cause. Add a matching error line so the operator can see why the verification failed.♻️ Proposed change
- return verify && paths.size > 0; + if (verify && paths.size > 0) { + logger.error(`verify: ${String(paths.size)} storage path(s) were never transferred`); + + return true; + } + + return false; };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer/import-verify.ts` around lines 135 - 145, Update reportUntransferredPaths so that when verify is true and paths is non-empty, it logs an error identifying the untransferred storage paths before returning true, matching the error reporting behavior of reportStorageOutcome. Preserve the existing warning details and return behavior for other cases.packages/cli/src/commands/data-transfer/sources/storage-transfer.ts (1)
215-235: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winRecord uploaded keys in
alreadyStoredto avoid re-uploading identical content.
alreadyStoredis built once from the target listing and is never updated. Two source objects with identical bytes produce the same content-hash key. The second object then re-uploads the same key. The result is correct, but the transfer pays for a duplicate upload on every duplicate blob.♻️ Proposed change
if (alreadyStored.get(key)?.size !== bytes.length) { const metadata: StorageMetadataRow = { contentType: entry.contentType, id: entry.path, sha256, size: bytes.length }; await uploadStorageBlob(context, bytes, metadata, options.keyPrefix, logger); + alreadyStored.set(key, { size: bytes.length }); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/data-transfer/sources/storage-transfer.ts` around lines 215 - 235, Update moveOneObject to add each successfully handled key to alreadyStored with its byte length after the upload-or-skip check, so later source objects with the same content hash avoid re-uploading while preserving the existing recordTransfer behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@apps/docs/src/content/blog/firestore-typed-values-and-the-export-that-isnt-json.mdx`:
- Around line 46-67: Update the Firestore dump-generation example described at
apps/docs/src/content/blog/firestore-typed-values-and-the-export-that-isnt-json.mdx:46-67
and apps/docs/src/content/docs/migrating/from-firebase.mdx:288-295 to emit REST
Value JSON rather than raw document._fieldsProto. Normalize each typed value,
including protobuf timestamps to RFC-3339 timestampValue strings, before writing
the dump, and keep both documented import examples consistent with that output.
In `@apps/docs/src/content/docs/migrating/from-convex.mdx`:
- Around line 304-307: Update the documented Convex export command in the
migration guide to include the file-storage option, so the resulting
snapshot.zip contains blobs that lunora import --with-storage can migrate.
In `@packages/cli/src/commands/convex-snapshot.ts`:
- Around line 162-169: Update the async iteration around createInterface in the
snapshot entry-stream handling to wrap consumption in a finally block that
closes the readline interface and destroys the underlying stream, including when
the consumer breaks or throws. Preserve the existing raw-line yielding and
undefined-stream return behavior.
In `@packages/cli/src/commands/data-transfer/import-rows.ts`:
- Around line 60-61: Update remapEnvelope to catch JSON.parse failures and
rethrow the same row-scoped LunoraError pattern used by wrapBareDocument,
including the provided lineNumber. Preserve successful envelope parsing and
ensure drainIntoBatcher receives the contextual error instead of a raw
SyntaxError.
In `@packages/cli/src/commands/data-transfer/import.ts`:
- Around line 282-324: Update runForeignStorageTransfer so only source.kind ===
"supabase" can proceed to the live Supabase listing and transfer; keep the
storageDir branch source-independent. For Firebase without --storage-dir, log
guidance to download the bucket with gcloud storage cp -r and provide
--storage-dir, then return undefined before reading Supabase environment
variables.
In `@packages/cli/src/commands/data-transfer/sources/auth.ts`:
- Around line 173-184: Update both auth mappers in
packages/cli/src/commands/data-transfer/sources/auth.ts:173-184 and
packages/cli/src/commands/data-transfer/sources/auth.ts:221-233 so each
account’s _id and id include the user ID, provider, and provider account ID. Use
providerAccountId for the Supabase identities and the resolved accountId for
Firebase providerUserInfo entries, preserving distinct rows for multiple
identities under the same provider.
In `@packages/cli/src/commands/data-transfer/sources/dump-directory.ts`:
- Around line 54-72: Update the mapping resolution in the dump-directory flow
around claimed, found, and readDumpFiles so every mapping entry with a file is
validated against directory entries independently of options.matches. Resolve
claimed basenames to matching files, ensure mapped files are present and
eligible for import, and fail or warn when a claimed file is missing or rejected
rather than silently falling back to discovery; preserve the existing table
association and ensure unresolved mapped tables remain visible to --verify.
In `@packages/cli/src/commands/data-transfer/sources/firebase.ts`:
- Around line 88-96: Update the timestampValue handling in the Firebase value
reshaping logic to preserve RFC-3339 fractional-second digits beyond
milliseconds instead of silently converting them through Date.parse. Keep the
original timestamp string for values with sub-millisecond precision while
retaining current validation and millisecond conversion for compatible
timestamps.
- Around line 201-264: Change the .ndjson handling in readFirestoreCollection
and readCollectionDocuments so the file is consumed as a stream and each
non-empty line is parsed and yielded immediately, rather than loading the full
text or document array. Preserve line trimming, 1-based line numbers, and the
existing invalid-JSON LunoraError message and cause. Keep the existing JSON-file
parsing paths unchanged.
- Around line 149-152: Update the wrapped-shape detection in the surrounding
decoder to use the string-valued name field when fallbackId is available, rather
than treating any object-valued fields property as wrapped. Preserve wrapped
decoding for REST documents and ensure bare community documents—including those
with a fields field—are decoded as complete records.
In `@packages/cli/src/commands/data-transfer/sources/scan.ts`:
- Around line 67-77: Update scanCsvTable to parse the CSV from a file read
stream and stop consuming input after collecting the SCAN_ROW_LIMIT sample,
avoiding readFile and whole-file buffering; follow the existing streaming
approach used by readSupabaseTable. While processing rows, replace the byColumn
array reconstruction with in-place push operations on the existing column
arrays.
In `@packages/cli/src/commands/data-transfer/sources/supabase.ts`:
- Around line 159-173: In the generator’s parser loop, adjust the try/catch
around parser reads so the catch covers only obtaining the next row, not the
subsequent yield to the consumer. Preserve the existing LunoraError context for
parser failures while allowing errors thrown by yield or consumer interaction to
propagate unchanged.
- Around line 121-142: Update the row-mapping logic around the id-column loop to
track explicitly whether the declared idColumn was encountered, rather than
using document["_id"] as the presence check. Set that flag only when column ===
idColumn, keep assigning the declared raw id to document["_id"], and make the
final missing-id error trigger when the flag is false so a non-id source column
named "_id" cannot satisfy the guard.
In `@packages/cli/src/commands/import/index.ts`:
- Around line 29-34: Update the option descriptions for “with-storage” and
“storage-dir” in the import command so they mention all supported Convex,
Supabase, and Firebase storage transfer sources, including that storage-dir also
supplies the Supabase transfer path.
In `@packages/cli/src/commands/zip-entry-stream.ts`:
- Around line 63-91: Update openZipEntryStream to compute CRC-32 for every
non-empty entry while streaming, including STORED and DEFLATED entries, and
compare the final checksum with entry.header.crc. Destroy the returned output
stream with a descriptive error when the checksum differs, while preserving
existing stream and decompression error propagation; add fixtures covering
corrupted stored and deflated archives.
In `@packages/codegen/src/discover-stale-migration-imports.ts`:
- Around line 15-18: Update the Firebase entry in MIGRATED_PLATFORMS to include
the firebase-admin package prefix alongside firebase and `@firebase/`. Ensure
platformFor recognizes firebase-admin and subpath imports such as
firebase-admin/auth as stale Firebase imports.
---
Nitpick comments:
In `@packages/cli/src/commands/data-transfer/import-verify.ts`:
- Around line 135-145: Update reportUntransferredPaths so that when verify is
true and paths is non-empty, it logs an error identifying the untransferred
storage paths before returning true, matching the error reporting behavior of
reportStorageOutcome. Preserve the existing warning details and return behavior
for other cases.
In `@packages/cli/src/commands/data-transfer/import.ts`:
- Around line 104-127: Update buildImportBody and its callers to accept the
transferred-path count, and emit ImportSummary.storage for foreign Supabase and
Firebase imports when --with-storage is used, even when storageIdMap is
undefined. Populate the storage summary with the number of transferred objects
while preserving the existing storage remap fields for sources that provide
storageIdMap, and keep the exported ImportSummary type consistent with both
paths.
- Around line 252-272: Update the runScan return type from Promise<unknown> to a
union of the concrete return types from scanStorageColumns, scanFirebaseDump,
and scanSupabaseDump, plus undefined for the default branch. Preserve the
existing scanner dispatch and caller’s undefined check.
In `@packages/cli/src/commands/data-transfer/sources/dump-directory.ts`:
- Around line 48-52: Update the directory-reading error handling around
readDirectory so the caught failure is retained and passed as the cause when
constructing LunoraError. Preserve the existing unreadable-directory message and
undefined-check behavior while allowing callers to distinguish failures such as
ENOENT, ENOTDIR, and EACCES.
In `@packages/cli/src/commands/data-transfer/sources/reshape.ts`:
- Around line 52-53: Merge EXPONENT_RE and EXPONENT_SPLIT_RE into a single /e/i
constant, remove the duplicate declaration, and update expandExponent and any
other references to use the retained EXPONENT_RE symbol.
In `@packages/cli/src/commands/data-transfer/sources/storage-progress.ts`:
- Around line 85-91: Refactor recordTransfer so the checkpoint directory is
created and the progress file is opened once per transfer run, then reuse the
append-capable file handle for each completed object. Preserve one JSON line per
entry and ensure the handle is closed after the run, rather than performing
mkdir and open/write/close operations for every object.
In `@packages/cli/src/commands/data-transfer/sources/storage-transfer.ts`:
- Around line 215-235: Update moveOneObject to add each successfully handled key
to alreadyStored with its byte length after the upload-or-skip check, so later
source objects with the same content hash avoid re-uploading while preserving
the existing recordTransfer behavior.
In `@packages/runtime/src/storage-admin-routes.ts`:
- Around line 177-180: Update the upload flow around storageUpload so the
SHA-256 digest is computed unconditionally for every request, rather than only
when expectedSha256 or expectedSize is provided, and pass that digest to
storageUpload for recording. If verification-only behavior is intentional
instead, narrow the nearby comment to state that the checksum is recorded only
for uploads requesting verification.
In `@packages/storage/src/create-storage.ts`:
- Around line 323-331: Validate uploadOptions.sha256 near
assertContentTypeAllowed before calling options.bucket.put: accept only a
64-character hexadecimal string or a 32-byte buffer, and otherwise raise the
module’s standard VALIDATION_ERROR while naming sha256. Keep valid digests
unchanged when constructing the put options.
- Around line 249-269: Update the Storage documentation package example to
provide a valid contentType whenever allowedContentTypes is configured, matching
assertContentTypeAllowed. Add a migration note explaining that uploads omitting
contentType now fail at runtime with VALIDATION_ERROR and must be updated to
declare an allowed content type.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: ce027a33-318d-4ef2-9b56-2fa50f48f886
⛔ Files ignored due to path filters (19)
api-snapshots/advisor.api.mdis excluded by none and included by noneapi-snapshots/cli.api.mdis excluded by none and included by noneapi-snapshots/codegen.api.mdis excluded by none and included by noneapi-snapshots/errors.api.mdis excluded by none and included by noneapi-snapshots/runtime.api.mdis excluded by none and included by noneapi-snapshots/storage.api.mdis excluded by none and included by noneexamples/blog/lunora/_generated/app.tsis excluded by!**/_generated/**and included by noneexamples/team-chat/lunora/_generated/app.tsis excluded by!**/_generated/**and included by nonepackages/advisor/__tests__/migration-stale-import.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/import-batcher.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/import-sources.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/import-storage.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/storage-path-index.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/zip-entry-stream.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/codegen/__tests__/fixtures/simple/expected/_generated/app.tsis excluded by!**/_generated/**,!**/__tests__/**and included bypackages/**packages/runtime/__tests__/storage-admin.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**plans/305-supabase-firebase-importers.mdis excluded by none and included by nonepnpm-lock.yamlis excluded by!**/pnpm-lock.yamland included by nonepnpm-workspace.yamlis excluded by none and included by none
📒 Files selected for processing (46)
apps/docs/src/content/blog/firestore-typed-values-and-the-export-that-isnt-json.mdxapps/docs/src/content/blog/migrate-convex-files-and-all.mdxapps/docs/src/content/blog/supabase-to-lunora-without-a-reshape-script.mdxapps/docs/src/content/docs/migrating/from-convex.mdxapps/docs/src/content/docs/migrating/from-firebase.mdxapps/docs/src/content/docs/migrating/from-supabase.mdxpackages/advisor/src/index.tspackages/advisor/src/lints/static/migration-stale-import.tspackages/advisor/src/stale-migration-imports.tspackages/advisor/src/types.tspackages/cli/package.jsonpackages/cli/src/commands/convex-snapshot.tspackages/cli/src/commands/data-transfer/import-rows.tspackages/cli/src/commands/data-transfer/import-source.tspackages/cli/src/commands/data-transfer/import-verify.tspackages/cli/src/commands/data-transfer/import.tspackages/cli/src/commands/data-transfer/index.tspackages/cli/src/commands/data-transfer/mapping-shapes.tspackages/cli/src/commands/data-transfer/shared.tspackages/cli/src/commands/data-transfer/sources/auth.tspackages/cli/src/commands/data-transfer/sources/dump-directory.tspackages/cli/src/commands/data-transfer/sources/firebase.tspackages/cli/src/commands/data-transfer/sources/mapping.tspackages/cli/src/commands/data-transfer/sources/reshape.tspackages/cli/src/commands/data-transfer/sources/scan.tspackages/cli/src/commands/data-transfer/sources/storage-progress.tspackages/cli/src/commands/data-transfer/sources/storage-transfer.tspackages/cli/src/commands/data-transfer/sources/supabase.tspackages/cli/src/commands/data-transfer/storage-blobs.tspackages/cli/src/commands/data-transfer/storage-mapping.tspackages/cli/src/commands/data-transfer/storage-path-index.tspackages/cli/src/commands/data-transfer/storage-remap.tspackages/cli/src/commands/import/handler.tspackages/cli/src/commands/import/index.tspackages/cli/src/commands/seed/handler.tspackages/cli/src/commands/zip-entry-stream.tspackages/codegen/src/advisor.tspackages/codegen/src/discover-stale-migration-imports.tspackages/codegen/src/emit-app.tspackages/codegen/src/ir.tspackages/codegen/src/run-codegen.tspackages/errors/src/catalog.tspackages/runtime/src/create-worker.tspackages/runtime/src/storage-admin-routes.tspackages/storage/src/create-storage.tspackages/storage/src/types.ts
Merging this PR will regress 1 benchmark
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | in-batch: single IN(...) query + id->doc re-projection |
812.6 µs | 950.9 µs | -14.54% |
| ⚡ | findMany — plain table (no scope) |
87.6 ms | 77.8 ms | +12.56% |
| ⚡ | count, no attributes |
67 µs | 60.5 µs | +10.8% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing feat/supabase-firebase-importers (e3ed7d0) with alpha (8e5d203)2
Footnotes
-
10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
alpha(262eccb) during the generation of this report, so 8e5d203 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Seven review findings, each verified against the code first. `supabase.ts` — the `try` wrapped the `yield`, so a generator resumes at its yield and an error thrown by the CONSUMER came back relabelled `<file> row N: …`, a parse-location message for something that never touched the parser. Reading and decoding a row stay attributed; the yield now sits outside. `supabase.ts` — a source column literally named `_id` landed in the target's reserved id field and satisfied the "did we find an id?" guard, so a table whose real `idColumn` was absent imported with the wrong value as its id — the dangling-FK failure that guard exists to prevent. The guard now tracks whether the id column was actually seen, and a non-id `_id` column is a named error rather than a silent collision. `firebase.ts` — a BARE document with a field called `fields` was read as the wrapped REST form, so every other field was dropped. `name` stays the reliable signal; without one, the tie-break is what `fields` holds — a map of name→`Value` (wrapped) versus a single one-key `Value` (bare) — and anything alongside it that is not wrapper metadata settles it as bare. `import.ts` — a Firebase source with `--with-storage` and no `--storage-dir` fell through to the Supabase remote path and was told to set `SUPABASE_URL`, for a migration with no Supabase in it. There is no remote Firebase download path, so it now says to pull the bucket first. `import-rows.ts` — a malformed NDJSON line threw a bare `Unexpected token` with no way to find it in a multi-GB file, while every other failure on that path names its line. `discover-stale-migration-imports.ts` — the matcher is exact-or-`prefix/`, so `firebase-admin` (the SDK a migrated backend actually imported) never matched `firebase`. Listed explicitly, with the test that was missing: the matcher had none, which is why the gap existed. `import/index.ts` — `--with-storage` still described itself as Convex-only. cli 1168, codegen 1068. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`convex-snapshot.ts` — a consumer that stops early (a `break`, a throw, an
import failing on row 12 of a million) left the readline interface and the
ZIP entry stream open, holding the archive's file handle for the rest of
the process. Closed in a `finally`.
`auth.ts` — both mappers keyed an account on `${user}:${provider}`, so a
user holding two identities at the SAME provider collapsed into one id:
the second silently replacing the first, or the import failing on a
duplicate. The provider account id is part of the key now, with a test
that imports two github identities for one user and asserts both survive.
`dump-directory.ts` — a mapping entry whose `file` is absent from the
directory, or present but filtered out (wrong extension), was dropped
without a word. The operator named that table expecting it to migrate, and
`--verify` still passed because it compares what WAS imported. Now a named
error listing each unresolved file and its table.
`scan.ts` — `--scan` read the WHOLE dump into memory to look at 200 rows
(`toLine` bounds what the parser emits, not what is read), and rebuilt the
per-column array on every value, so sampling N rows cost O(N²) copies. It
streams, stops at the sample size, destroys the handle, and accumulates in
place.
`firebase.ts` — the `timestampValue` truncation to milliseconds is now
stated where it happens and in the migration guide. Keeping the RFC-3339
string for sub-millisecond rows was the suggested fix and is worse: the
column would be number for most rows and string for a few, which no schema
describes. Firestore stores microseconds; a Lunora timestamp is
milliseconds; an app that needs the rest should carry it in its own column.
cli 1169.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The documented Firestore dump script reads `document._fieldsProto`, and
that is the Admin SDK's own representation — *almost* the REST encoding,
not exactly it. `Timestamp.toProto()` returns
`{ timestampValue: { seconds, nanos } }` where REST returns an RFC-3339
string, and bytes arrive as a `Buffer` rather than base64.
The decoder accepted only the REST spelling, so `Date.parse` ran against
an object, returned NaN, and threw "is not an RFC-3339 timestamp" — on the
first document with a `createdAt`. The fifteen-line script the blog post
advertises did not produce a dump this importer could read.
Both spellings are accepted now, with a test that imports the
`_fieldsProto` shape and asserts the decoded millisecond value and base64.
The blog post and the guide say which is which rather than implying the
two are the same encoding.
Also: the guide's `snapshot.zip` line omitted `--include-file-storage`, so
following it produced an archive with no blobs — and then `--with-storage`
had nothing to migrate, while the CLI's own `--verify` error tells you the
export needed that flag. The directory command already had it; the ZIP one
now matches.
Finally, `.ndjson` collections stream a line at a time instead of reading
the file into a string and then building an array of every document — two
full copies of a collection that can be tens of gigabytes. The `.json`
container shapes are single JSON values and still parse whole.
cli 1170.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lint follow-up to the previous commit: the added ternaries pushed `decodeValue` past the cognitive-complexity limit, and each branch had grown a paragraph of context that reads better next to the code it explains than inside a fifteen-way dispatch. No behaviour change — cli 1170. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The archive already carries the checksum its writer computed, and nothing compared it. A corrupted or truncated entry imported as data — silently, with row counts that still matched, so `--verify` reported nothing either: it compares counts, not contents. Both paths are checked. STORED bytes are the entry's contents already; DEFLATED ones are checked after inflating, which is what the header's CRC describes. `zlib.crc32` is the same polynomial ZIP uses (Node ≥22.2, and this repo requires ^22.15), so this is a comparison rather than a reimplementation. The test corrupts a good archive's recorded checksum and asserts the stream rejects rather than yielding the bytes. cli 1171. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Presented as literal CLI output, so it should be. The real message joins both clauses with an em dash; the post had a comma in the second one. Checked the rest of the three posts against the code rather than reading them for tone. Accurate: the Postgres NULL-vs-empty-string distinction (`castPostgresCsv`), the digit-loss refusal, `integerValue` staying a string outside the safe range (`Number.isSafeInteger(parsed) ? parsed : raw`), SHA-256 content keying, the 32 MiB verified-upload cap, the signed-PUT fallback, and deleting an object that fails verification. `--pg-url` appears only as a rejected alternative, not a claimed flag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/zip-entry-stream.ts`:
- Around line 117-125: Validate zero-sized entries before the existing CRC/read
bypass in the zip-entry stream flow, rejecting any entry whose declared
uncompressedSize is nonzero or whose header.crc is nonzero. Preserve the empty
result only when all zero-size metadata is consistent, and add a test that
modifies a non-empty entry to declare zero compressed bytes.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 23831844-9f54-49a6-b0f5-4fd88f98ae30
⛔ Files ignored due to path filters (3)
packages/cli/__tests__/commands/import-sources.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/zip-entry-stream.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/codegen/__tests__/discover-stale-migration-imports.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (15)
apps/docs/src/content/blog/firestore-typed-values-and-the-export-that-isnt-json.mdxapps/docs/src/content/blog/supabase-to-lunora-without-a-reshape-script.mdxapps/docs/src/content/docs/migrating/from-convex.mdxapps/docs/src/content/docs/migrating/from-firebase.mdxpackages/cli/src/commands/convex-snapshot.tspackages/cli/src/commands/data-transfer/import-rows.tspackages/cli/src/commands/data-transfer/import.tspackages/cli/src/commands/data-transfer/sources/auth.tspackages/cli/src/commands/data-transfer/sources/dump-directory.tspackages/cli/src/commands/data-transfer/sources/firebase.tspackages/cli/src/commands/data-transfer/sources/scan.tspackages/cli/src/commands/data-transfer/sources/supabase.tspackages/cli/src/commands/import/index.tspackages/cli/src/commands/zip-entry-stream.tspackages/codegen/src/discover-stale-migration-imports.ts
🚧 Files skipped from review as they are similar to previous changes (12)
- apps/docs/src/content/docs/migrating/from-convex.mdx
- packages/cli/src/commands/convex-snapshot.ts
- packages/codegen/src/discover-stale-migration-imports.ts
- packages/cli/src/commands/import/index.ts
- packages/cli/src/commands/data-transfer/import-rows.ts
- packages/cli/src/commands/data-transfer/sources/supabase.ts
- packages/cli/src/commands/data-transfer/sources/auth.ts
- apps/docs/src/content/docs/migrating/from-firebase.mdx
- packages/cli/src/commands/data-transfer/sources/scan.ts
- apps/docs/src/content/blog/supabase-to-lunora-without-a-reshape-script.mdx
- packages/cli/src/commands/data-transfer/import.ts
- packages/cli/src/commands/data-transfer/sources/firebase.ts
Follow-up to the CRC check in 853122b, which left a hole in itself: the `compressedSize === 0` early return ran BEFORE any verification, so a header claiming zero compressed bytes skipped the checksum entirely. The entry read as empty and imported as no rows — silent wrong data, which is the case the CRC exists to stop, reached by the one path that never consults it. An empty entry is empty in all three fields, so the early return now requires that: zero compressed, zero uncompressed, zero CRC. Anything else claiming zero compression is a corrupt header and says so. The genuinely-empty case still returns `undefined` — its test is unchanged and still passes. The new one takes a good archive and rewrites `compressedSize` to 0. cli 1172. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Builds on the Convex blob migration merged in #354, reusing its id
preservation, content-hash keys and verification machinery.
lunora import --from supabaseand--from firebasenow take a dump the wholeway: rows, auth, and files, with the same id preservation, parity check and
dangling-reference reporting the Convex path already had.
What it adds
Sources. Supabase Postgres CSV and Firestore export JSON both decode into
the one
{ table, doc }envelope the importer already speaks. The CSV castdistinguishes an unquoted empty field (NULL) from a quoted
""(empty string)and from
\N(text-format NULL) — collapsing those three is silent data loss.Firestore's typed-value encoding (
integerValueas a string,timestampValue,mapValue,referenceValue,geoPointValue,bytesValue) decodes to plainJSON.
Mapping.
lunora/import-<source>.jsonstates which file feeds which table,which column is the id, which columns hold storage paths, and any declared
reshape.
--scanproposes one by sampling the dump. A mapping that fails toparse throws naming the key rather than degrading to "no mapping" — a dropped
mapping turns a declared conversion into a silent pass-through.
Auth. Supabase
auth.users+auth.identitiesand a Firebaseauth:exportbecome better-auth
user/accountrows. Password hashes and reset tokens arenever emitted; users sign in again.
Storage. Supabase objects transfer live over the Storage REST API with a
service-role key read from the environment (never a flag — a flag is visible in
the process table). Firebase objects come from a
gcloud storage cp -rdirectory, because
gcloudalready owns Google's auth. Both then take theidentical path into R2: content-hash key, checksum-verified upload,
skip-if-present, delete-on-mismatch. The transfer is resumable — every
object is checkpointed, so a run that dies at object 40,000 of 50,000 resumes
without re-downloading what already moved. A partial transfer deliberately does
not import rows: every path column would point at an object that is not there
yet.
Advisor. A static lint flags a migrated-away platform's SDK still imported
under
lunora/.Docs. Rewritten data-migration steps in the Supabase and Firebase guides.
Behaviour changes worth a reviewer's attention
success while emitting credential columns as table rows, or import a table
with no
_idat all. Both reproduced before the fix; both now fail closed.storageUploadpassessha256through toR2, so
list()/head()report a real checksum instead of degrading everydownstream integrity check to a size comparison.
while the ZIP import inflated each entry whole into a string and split it into
an array of every line — so the same export succeeded as a directory and ran
the CLI out of heap as an archive. It now reads the entry's byte range through
zlib.bucket-relative, or a
getPublicUrl()/createSignedUrl()URL. A bare filenametwo buckets both carry stays unresolved on purpose: guessing would rewrite rows
at the wrong object.
ImportCommandResult.bodyisImportSummary | undefinedinstead ofunknown(the one api-snapshot change).Verification
lint:types,lint:eslint,lint:prettier,lint:package-json,api:checkand
dist:checkall green. Suites: cli 1168, storage 109, runtime 845, advisor467, codegen 1054, seed 79, studio 1012.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
lunora importsupport for Firebase and Supabase exports, including CSV, Firestore JSON, authentication data, and storage files.Documentation