Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/pages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,6 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: actions/configure-pages@v5
with:
enablement: true
- uses: actions/upload-pages-artifact@v3
with:
path: site
Expand Down
56 changes: 31 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ stores sync durable state through Git tree commits.

```text
App code
-> GitDB DataSource / Repository
-> GitDB database + table handles
-> Local SQL engine + equality indexes
-> Manifest, mutation log, page snapshots, compaction
-> GitHub repository for durable history and audit
Expand All @@ -26,7 +26,7 @@ Use GitDB when you want:

- A database repository per project, for example `my-app-db`
- Local query execution without per-query network round trips
- A TypeORM-style `DataSource` and repository API included in the package
- GitDB-native SQL and table APIs without a third-party adapter layer
- Human-inspectable plaintext snapshots for public demos and reviewable data
- Encrypted manifest and mutation logs for private local data
- Auditable Git commits for agents, demos, content tools, config tools, and
Expand All @@ -40,8 +40,8 @@ multi-node coordination.

| Area | Current behavior |
| --- | --- |
| App API | `createGitDbDataSource`, `defineEntity`, typed repositories |
| ORM | `save`, `insert`, `insertMany`, `saveMany`, `find`, `findOne`, `delete` |
| App API | `GitDb.open`, `openGitDb`, `defineTable`, typed table handles |
| Table API | `insert`, `insertMany`, `upsert`, `upsertMany`, `select`, `first`, `deleteWhere` |
| SQL engine | `CREATE TABLE`, `INSERT`, `DELETE`, `SELECT`, joins, grouping, ordering, aggregates |
| Indexes | Rebuildable primary/equality indexes for simple equality lookup |
| Storage | Local plaintext, local encrypted, GitHub plaintext, GitHub encrypted |
Expand Down Expand Up @@ -117,42 +117,47 @@ Install:
npm install @3xhaust/gitdb
```

Use the first-party repository API:
Use the first-party table API:

```ts
import { LocalPlaintextStore, createGitDbDataSource, defineEntity } from "@3xhaust/gitdb"
import { defineTable, GitDb, LocalPlaintextStore } from "@3xhaust/gitdb"
import { z } from "zod"

type Person = {
readonly id: string
readonly name: string
readonly team_id: string
}
const PersonRow = z.object({
id: z.string(),
name: z.string(),
team_id: z.string(),
})

const PersonEntity = defineEntity<Person>({
const Person = defineTable({
columns: { id: "STRING", name: "STRING", team_id: "STRING" },
indexes: [{ columns: ["team_id"], name: "people_team_id_idx" }],
name: "people",
primaryKey: "id",
tableName: "people",
row: PersonRow,
})

const dataSource = await createGitDbDataSource({
entities: [PersonEntity],
const db = await GitDb.open({
store: new LocalPlaintextStore({ root: ".gitdb" }),
synchronize: true,
syncSchema: true,
tables: [Person],
})

const people = dataSource.getRepository(PersonEntity)
const people = db.table(Person)
await people.insertMany([
{ id: "p1", name: "Lin", team_id: "storage" },
{ id: "p2", name: "Ada", team_id: "runtime" },
])
await people.saveMany([{ id: "p2", name: "Ada Lovelace", team_id: "runtime" }])
await people.upsertMany([{ id: "p2", name: "Ada Lovelace", team_id: "runtime" }])

const storagePeople = await people.find({ where: { team_id: "storage" } })
const storagePeople = await people.select({ team_id: "storage" })
const joined = await db.query(
"SELECT people.name FROM people WHERE people.team_id = 'storage'",
)
```

Secondary indexes are equality-only metadata on `defineEntity`. The runtime uses
rebuilt equality indexes for simple `find({ where })` and `SELECT * ... WHERE`
Secondary indexes are equality-only metadata on `defineTable`. The runtime uses
rebuilt equality indexes for simple `select({ ... })` and `SELECT * ... WHERE`
lookups when the query shape is safe; other SQL falls back to the local SQL
engine.

Expand Down Expand Up @@ -216,12 +221,13 @@ Run the bundled example:
corepack pnpm example
```

This builds the package and runs the plaintext API example plus two encrypted
API examples:
This builds the package and runs the plaintext and encrypted GitHub-backed API
examples against a separate database repository. Set `GITDB_GITHUB_OWNER`,
`GITDB_GITHUB_TOKEN`, and `GITDB_KEY` first; `GITDB_GITHUB_REPO` defaults to
`gitdb-example-db` and GitDB creates that repo when it is missing:

- `examples/api-local`
- `examples/api-plaintext`
- `examples/api-encrypted`
- `examples/api-encrypted-reopen`

Run local benchmarks:

Expand Down
1 change: 1 addition & 0 deletions biome.json
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
"!.gitdb-example-public",
"!.claude",
"!.omc",
"!.omo",
"!.omx"
]
},
Expand Down
52 changes: 27 additions & 25 deletions docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,67 +10,69 @@ npm install @3xhaust/gitdb

GitDB is ESM-only and publishes types from `dist/src/index.d.ts`.

## Data Source
## GitDB

```ts
import { LocalPlaintextStore, createGitDbDataSource, defineEntity } from "@3xhaust/gitdb"
import { defineTable, GitDb, LocalPlaintextStore } from "@3xhaust/gitdb"
import { z } from "zod"

type Person = {
readonly id: string
readonly name: string
readonly team_id: string
}
const PersonRow = z.object({
id: z.string(),
name: z.string(),
team_id: z.string(),
})

const Person = defineEntity<Person>({
const Person = defineTable({
columns: { id: "STRING", name: "STRING", team_id: "STRING" },
indexes: [{ columns: ["team_id"], name: "people_team_id_idx" }],
name: "people",
primaryKey: "id",
tableName: "people",
row: PersonRow,
})

const dataSource = await createGitDbDataSource({
entities: [Person],
const db = await GitDb.open({
store: new LocalPlaintextStore({ root: ".gitdb" }),
synchronize: true,
syncSchema: true,
tables: [Person],
})
```

`synchronize: true` creates missing tables from entity metadata. It does not run
`syncSchema: true` creates missing tables from table metadata. It does not run
destructive migrations.

## Repository
## Table Handles

```ts
const people = dataSource.getRepository(Person)
const people = db.table(Person)

await people.insert({ id: "p1", name: "Lin", team_id: "storage" })
await people.insertMany([
{ id: "p2", name: "Ada", team_id: "runtime" },
{ id: "p3", name: "Grace", team_id: "storage" },
])

await people.saveMany([{ id: "p2", name: "Ada Lovelace", team_id: "runtime" }])
await people.upsertMany([{ id: "p2", name: "Ada Lovelace", team_id: "runtime" }])

const storagePeople = await people.find({ where: { team_id: "storage" } })
const ada = await people.findOne({ where: { id: "p2" } })
const deleted = await people.delete({ id: "p1" })
const storagePeople = await people.select({ team_id: "storage" })
const ada = await people.first({ id: "p2" })
const deleted = await people.deleteWhere({ id: "p1" })
```

`insertMany()` and `saveMany()` share one transaction boundary. Prefer them over
repeated single-row `save()` calls when loading batches.
`insertMany()` and `upsertMany()` share one transaction boundary. Prefer them
over repeated single-row calls when loading batches.

## Indexes

Indexes are equality-only and rebuildable. The entity metadata records intent:
Indexes are equality-only and rebuildable. The table metadata records intent:

```ts
indexes: [{ columns: ["team_id"], name: "people_team_id_idx" }]
```

The runtime maintains primary/equality indexes from committed rows. Simple
repository `find({ where })` calls and safe `SELECT * FROM table WHERE col =
literal` queries use the index path. Stale or missing index snapshot files are
ignored because indexes are derived data.
table `select({ ... })` calls and safe `SELECT * FROM table WHERE col = literal`
queries use the index path. Stale or missing index snapshot files are ignored
because indexes are derived data.

## Engine

Expand Down
10 changes: 5 additions & 5 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ durable history and audit.
## Runtime Layers

1. **First-party API**
- `createGitDbDataSource`
- `defineEntity`
- `GitDbRepository`
- `insert`, `insertMany`, `saveMany`, `find`, `findOne`, `delete`
- `GitDb.open`
- `openGitDb`
- `defineTable`
- `insert`, `insertMany`, `upsertMany`, `select`, `first`, `deleteWhere`

2. **SQL engine**
- Local SQL execution with joins, grouping, ordering, and aggregates
Expand All @@ -24,7 +24,7 @@ durable history and audit.
- Additive batch commit boundary
- Optional lock, snapshot, compaction, and GitHub tree-commit capabilities

4. **Repository storage**
4. **Git storage**
- Local plaintext for public, inspectable page snapshots
- Local encrypted for opaque manifest and log files
- GitHub plaintext/encrypted for remote durable history
Expand Down
34 changes: 17 additions & 17 deletions docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,11 @@ Runtime metadata:
| --- | ---: | ---: | ---: | ---: | ---: |
| local plaintext transaction batch | 250 | 1438.67 | 173.77 | 5.48 | 66.35 |
| local encrypted transaction batch | 250 | 2173.42 | 115.03 | 0.73 | 41.65 |
| orm save() single transactions | 250 | 12304.65 | 20.32 | 0.31 | 107.08 |
| orm insert() single transactions | 250 | 8435.10 | 29.64 | 0.24 | 53.70 |
| orm insertMany bulk transaction | 250 | 1049.41 | 238.23 | 0.13 | 55.53 |
| orm saveMany bulk transaction | 250 | 2409.24 | 103.77 | 0.08 | 76.86 |
| orm indexed equality lookup | 250 | 920.12 | 271.70 | 0.08 | 45.93 |
| table upsert() single transactions | 250 | 12304.65 | 20.32 | 0.31 | 107.08 |
| table insert() single transactions | 250 | 8435.10 | 29.64 | 0.24 | 53.70 |
| table insertMany bulk transaction | 250 | 1049.41 | 238.23 | 0.13 | 55.53 |
| table upsertMany bulk transaction | 250 | 2409.24 | 103.77 | 0.08 | 76.86 |
| table indexed equality lookup | 250 | 920.12 | 271.70 | 0.08 | 45.93 |
| local plaintext reopen checkpoint | 250 | 1142.73 | 218.77 | 0.74 | 36.38 |
| local plaintext compaction | 250 | 105.46 | 2370.68 | 0.67 | 24.41 |

Expand All @@ -77,10 +77,10 @@ The same-row previous runtime baseline is stored in
| --- | ---: | ---: | ---: | ---: | ---: |
| local plaintext transaction batch | 173.14 | 173.77 | +0.36% | +0.36% | +83.37% |
| local encrypted transaction batch | 328.09 | 115.03 | -64.94% | -185.23% | +83.04% |
| orm indexed equality lookup | 46.49 | 271.70 | +484.44% | +82.89% | +93.88% |
| table indexed equality lookup | 46.49 | 271.70 | +484.44% | +82.89% | +93.88% |

The headline improvement is the first-party ORM path: indexed lookup plus bulk
write setup now runs at 271.70 writes/s, a 5.84x speedup over the previous ORM
The headline improvement is the first-party indexed table API path: indexed lookup plus bulk
write setup now runs at 271.70 writes/s, a 5.84x speedup over the previous indexed table API
baseline. Local plaintext write throughput is roughly flat with the previous
raw baseline in this run, while query latency is still much lower. Local encrypted
write throughput is currently slower than the older baseline because encryption
Expand All @@ -92,22 +92,22 @@ encrypted compaction remains future work.
The single-row APIs remain in the benchmark because they show the cost of
committing every row independently:

| Path | Writes/s | Compared with save() |
| Path | Writes/s | Compared with upsert() |
| --- | ---: | ---: |
| `save()` per row | 20.32 | 1.00x |
| `upsert()` per row | 20.32 | 1.00x |
| `insert()` per row | 29.64 | 1.46x |
| `insertMany()` | 238.23 | 11.73x |
| `saveMany()` | 103.77 | 5.11x |
| `upsertMany()` | 103.77 | 5.11x |

For application code, prefer `insertMany()` and `saveMany()` when a workflow can
For application code, prefer `insertMany()` and `upsertMany()` when a workflow can
commit a logical batch as one transaction.

## How To Read The Numbers

- `writeMs` is the time spent on the scenario's write or maintenance operation.
- `writesPerSecond` normalizes `rows / writeMs`.
- `Query ms` is a join for engine scenarios and an indexed equality lookup for
ORM/index scenarios.
indexed table API/index scenarios.
- `reopenMs` measures a fresh runtime opening the committed store and running
the validation query.
- The benchmark runner emits JSON metadata for Node, platform, CPU, repeat
Expand All @@ -119,10 +119,10 @@ commit a logical batch as one transaction.

- local plaintext transaction batch
- local encrypted transaction batch
- ORM single-row `save()`
- ORM single-row `insert()`
- ORM `insertMany()`
- ORM `saveMany()`
- indexed table API single-row `upsert()`
- indexed table API single-row `insert()`
- indexed table API `insertMany()`
- indexed table API `upsertMany()`
- indexed equality lookup
- reopen checkpoint
- compaction
Expand Down
2 changes: 1 addition & 1 deletion docs/BENCHMARK_BASELINE.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
},
{
"joinMs": 1.3,
"label": "orm local plaintext indexed lookup",
"label": "table indexed equality lookup",
"reopenMs": 136.29,
"rows": 250,
"writeMs": 5377.34,
Expand Down
13 changes: 7 additions & 6 deletions docs/MIGRATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,19 @@ contract.
The root package export is the stable consumer surface. Treat anything outside
this list as internal unless a future release documents it here.

- `createGitDbDataSource`
- `defineEntity`
- `GitDbDataSource`
- `GitDbRepository`
- `GitDbEntityIndexDefinition`
- `GitDb`
- `openGitDb`
- `defineTable`
- `GitDbTable`
- `GitDbTableIndexDefinition`
- `GitDbInsertResult`
- `GitDbSaveManyResult`
- `GitDbUpsertResult`
- `GitDbEngine`
- `GitDbDurabilityMode`
- `GitDbEngineOptions`
- `LocalPlaintextStore`
- `LocalEncryptedStore`
- `GitHubPlaintextStore`
- `GitHubEncryptedStore`
- `GitDbStore`
- `GitDbStoreV2`
Expand Down
Loading