Skip to content

feat: Add script for migrating data from CockroachDB to SQLite - #879

Open
KallynGowdy wants to merge 1 commit into
developfrom
claude/cockroachdb-sqlite-migration-kmkj7e
Open

KallynGowdy wants to merge 1 commit into
developfrom
claude/cockroachdb-sqlite-migration-kmkj7e

Conversation

@KallynGowdy

Copy link
Copy Markdown
Member

Adds npm run migrate:cockroachdb-to-sqlite, which copies every row from
a CockroachDB database into a SQLite database. Both databases are expected
to already have all of their migrations applied - the script only moves
data and never touches DDL or _prisma_migrations.

The copy plan is derived at runtime from the DMMF of both generated Prisma
clients, so all 68 models are covered without a hand-written mapping and
the script stays in sync as the schemas change. It applies the differences
between auth.prisma and auth.sqlite.prisma:

  • DateTime columns become epoch milliseconds (Decimal), matching
    convertToMillis() in aux-backend/prisma/Utils.ts.
  • String[] columns become JSON arrays.
  • Json nulls are written as Prisma.JsonNull/Prisma.DbNull depending
    on whether the target column is nullable.
  • BranchUpdate.updated maps to BranchUpdate.updatedAt.
  • Rows whose value is null in a column the SQLite schema requires (i.e.
    Invoice.subscriptionId) are skipped and reported.

Rows are read with keyset pagination ordered by primary key so large
tables don't slow down as the copy progresses, and written with
createMany in chunks sized to stay under SQLite's parameter limit.
Models are inserted in dependency order with PRAGMA foreign_keys = OFF,
since the schema contains two genuine foreign key cycles
(User <-> LoginRequest and Record -> FinancialAccount -> ContractRecord).
PRAGMA foreign_key_check runs at the end and a non-empty result exits
non-zero.

The script refuses to run against a non-empty target unless --truncate
or --skip-duplicates is given, and supports --dry-run, --only,
--exclude, --batch-size, and --yes.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01GRQLzHumoQT9P6zwiNQFyV

Adds `npm run migrate:cockroachdb-to-sqlite`, which copies every row from
a CockroachDB database into a SQLite database. Both databases are expected
to already have all of their migrations applied - the script only moves
data and never touches DDL or `_prisma_migrations`.

The copy plan is derived at runtime from the DMMF of both generated Prisma
clients, so all 68 models are covered without a hand-written mapping and
the script stays in sync as the schemas change. It applies the differences
between auth.prisma and auth.sqlite.prisma:

-   `DateTime` columns become epoch milliseconds (`Decimal`), matching
    `convertToMillis()` in aux-backend/prisma/Utils.ts.
-   `String[]` columns become JSON arrays.
-   Json nulls are written as `Prisma.JsonNull`/`Prisma.DbNull` depending
    on whether the target column is nullable.
-   `BranchUpdate.updated` maps to `BranchUpdate.updatedAt`.
-   Rows whose value is null in a column the SQLite schema requires (i.e.
    `Invoice.subscriptionId`) are skipped and reported.

Rows are read with keyset pagination ordered by primary key so large
tables don't slow down as the copy progresses, and written with
`createMany` in chunks sized to stay under SQLite's parameter limit.
Models are inserted in dependency order with `PRAGMA foreign_keys = OFF`,
since the schema contains two genuine foreign key cycles
(User <-> LoginRequest and Record -> FinancialAccount -> ContractRecord).
`PRAGMA foreign_key_check` runs at the end and a non-empty result exits
non-zero.

The script refuses to run against a non-empty target unless `--truncate`
or `--skip-duplicates` is given, and supports `--dry-run`, `--only`,
`--exclude`, `--batch-size`, and `--yes`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRQLzHumoQT9P6zwiNQFyV
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants