feat: Add script for migrating data from CockroachDB to SQLite - #879
Open
KallynGowdy wants to merge 1 commit into
Open
KallynGowdy wants to merge 1 commit into
KallynGowdy wants to merge 1 commit into
Conversation
Adds `npm run migrate:cockroachdb-to-sqlite`, which copies every row from
a CockroachDB database into a SQLite database. Both databases are expected
to already have all of their migrations applied - the script only moves
data and never touches DDL or `_prisma_migrations`.
The copy plan is derived at runtime from the DMMF of both generated Prisma
clients, so all 68 models are covered without a hand-written mapping and
the script stays in sync as the schemas change. It applies the differences
between auth.prisma and auth.sqlite.prisma:
- `DateTime` columns become epoch milliseconds (`Decimal`), matching
`convertToMillis()` in aux-backend/prisma/Utils.ts.
- `String[]` columns become JSON arrays.
- Json nulls are written as `Prisma.JsonNull`/`Prisma.DbNull` depending
on whether the target column is nullable.
- `BranchUpdate.updated` maps to `BranchUpdate.updatedAt`.
- Rows whose value is null in a column the SQLite schema requires (i.e.
`Invoice.subscriptionId`) are skipped and reported.
Rows are read with keyset pagination ordered by primary key so large
tables don't slow down as the copy progresses, and written with
`createMany` in chunks sized to stay under SQLite's parameter limit.
Models are inserted in dependency order with `PRAGMA foreign_keys = OFF`,
since the schema contains two genuine foreign key cycles
(User <-> LoginRequest and Record -> FinancialAccount -> ContractRecord).
`PRAGMA foreign_key_check` runs at the end and a non-empty result exits
non-zero.
The script refuses to run against a non-empty target unless `--truncate`
or `--skip-duplicates` is given, and supports `--dry-run`, `--only`,
`--exclude`, `--batch-size`, and `--yes`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRQLzHumoQT9P6zwiNQFyV
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
npm run migrate:cockroachdb-to-sqlite, which copies every row froma CockroachDB database into a SQLite database. Both databases are expected
to already have all of their migrations applied - the script only moves
data and never touches DDL or
_prisma_migrations.The copy plan is derived at runtime from the DMMF of both generated Prisma
clients, so all 68 models are covered without a hand-written mapping and
the script stays in sync as the schemas change. It applies the differences
between auth.prisma and auth.sqlite.prisma:
DateTimecolumns become epoch milliseconds (Decimal), matchingconvertToMillis()in aux-backend/prisma/Utils.ts.String[]columns become JSON arrays.Prisma.JsonNull/Prisma.DbNulldependingon whether the target column is nullable.
BranchUpdate.updatedmaps toBranchUpdate.updatedAt.Invoice.subscriptionId) are skipped and reported.Rows are read with keyset pagination ordered by primary key so large
tables don't slow down as the copy progresses, and written with
createManyin chunks sized to stay under SQLite's parameter limit.Models are inserted in dependency order with
PRAGMA foreign_keys = OFF,since the schema contains two genuine foreign key cycles
(User <-> LoginRequest and Record -> FinancialAccount -> ContractRecord).
PRAGMA foreign_key_checkruns at the end and a non-empty result exitsnon-zero.
The script refuses to run against a non-empty target unless
--truncateor
--skip-duplicatesis given, and supports--dry-run,--only,--exclude,--batch-size, and--yes.Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01GRQLzHumoQT9P6zwiNQFyV