Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ Each flag can also be set with an environment variable, which can come from a [.
| `ONEBASE_ADMIN_KEY` | Dashboard and deploy key (default: generated into `<data>/admin.key`) |
| `ONEBASE_DEPLOY_URL`, `ONEBASE_DEPLOY_KEY` | `deploy` and `env`: the server, and its admin key. Keep them in `.env.local`. |
| `ONEBASE_QUERY_TIMEOUT`, `ONEBASE_MUTATION_TIMEOUT`, `ONEBASE_ACTION_TIMEOUT` | How long a call may run (default `10s`, `10s`, `10m`; `off` for no limit). See [Time limits](./functions.md#time-limits). |
| `ONEBASE_WATCHDOG` | How long the server's main thread may be stuck before the process stops itself for its supervisor to restart (default `30s`; `off` to disable). See [Time limits](./functions.md#time-limits) |
| `ONEBASE_MAX_READS`, `ONEBASE_MAX_WRITES`, `ONEBASE_MAX_WRITE_MB`, `ONEBASE_MAX_DOCUMENT_MB` | How much one query or mutation may read and write (default `256000`, `128000`, `128`, `16`; `off` for no limit). See [Read and write limits](./functions.md#read-and-write-limits). |
| `ONEBASE_QUERY_CACHE_MB` | Memory for repeated query results on SQLite (default `64`; `0` to run every read). See [Repeated reads](./functions.md#repeated-reads). |
| `ONEBASE_VECTOR_MEMORY_MB` | Memory for SQLite's vector indexes (default `2048`; `0` to search them in SQL). See [Vector search](./vector-search.md#memory-and-threads-on-sqlite). |
Expand Down
2 changes: 1 addition & 1 deletion docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Deploying 5 source file(s) (6.0 KB) and 1 static file(s) from public/ (7.7 KB) t
What happens:

1. **On your machine:** `codegen` and `doctor` run, and doctor errors stop the deploy. Your functions and the npm packages they import are bundled into one module. The `public/` folder is included (pick another folder with `--public ./dist`).
2. **On the server:** the new code is loaded, and auth config and crons are checked. When `schema.ts` changed, every document is checked against the new schema, in the same transaction that updates the indexes, like a Convex push. If any check fails, nothing changes and the old code keeps running.
2. **On the server:** the new code is loaded, and auth config and crons are checked. When `schema.ts` changed, every document is checked against the new schema, in the same transaction that updates the indexes, like a Convex push. If any check fails, nothing changes and the old code keeps running. On SQLite that check doesn't hold up the server: documents are read 500 at a time before the transaction, with requests served in between, and the transaction only looks again at documents written meanwhile.
3. **The swap:** new calls use the new code, and calls already running finish with the old code. The replaced deployment's module is dropped from the module cache, so its memory is freed once those calls finish. Connected clients stay connected, and their live queries re-run on the new code.

Two more forms:
Expand Down
2 changes: 2 additions & 0 deletions docs/functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,8 @@ An action's worker is stopped when its time is up, even in the middle of `while

Queries and mutations run on the main thread, and JavaScript can't stop a function in the middle of an `await`. So a query or mutation keeps running until it next yields. Anything it then tries fails with the same error, so it can't read or write any more, call other functions, or schedule them. Code that never yields, such as `while (true) {}`, can't be stopped this way and blocks the whole server. (A background re-run on a [query worker](#live-queries-on-worker-threads) is the exception: its worker is stopped.) The same goes for actions with `ONEBASE_ACTION_WORKERS=0`.

**The watchdog.** For that last case, `onebase serve` runs a second thread that only watches whether the main one is still turning. When it has been stuck for 30 seconds, the process prints which function started last and stops itself, so whatever runs it (systemd, Docker, Fly) starts a fresh one: a server that would have hung until someone noticed is back in seconds. It stops with `SIGKILL`, since a stuck server can't shut down politely; nothing committed is lost. `ONEBASE_WATCHDOG` sets the wait (`2m`) or turns it off (`off`). Under `onebase dev` the server stops and comes back the next time you save a file.

The limit counts from when the call gets its database snapshot (a query), the write lock (a mutation) or a worker (an action), not from when it was queued. A mutation called from a mutation shares its caller's limit; one called from an action gets its own.

Change the limits with `ONEBASE_QUERY_TIMEOUT`, `ONEBASE_MUTATION_TIMEOUT` and `ONEBASE_ACTION_TIMEOUT`, e.g. `30s`, `2m`, or `off`. Keep mutations short: while one runs, every other write waits for it.
Expand Down
9 changes: 9 additions & 0 deletions src/app.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { extname, join, resolve, sep } from "node:path";
import { RateLimiter, serve, type AssetResolver, type RateLimit } from "./engine/http";
import { startBackups, type BackupOptions } from "./backup";
import { createDriver } from "./engine/driver";
import { startWatchdog } from "./engine/watchdog";
import { LocalBlobs, parseS3Url, S3Blobs } from "./engine/files";
import type { AuthConfig } from "./engine/jwt";
import { consoleLogger, DEFAULT_TIME_LIMITS, formatMs, jsonLogger, Runtime, type Authenticate, type Modules, type TimeLimits } from "./engine/runtime";
Expand All @@ -28,6 +29,11 @@ export type AppOptions = {
dashboard?: boolean;
/** Serve the dashboard and admin API only when the request's Host is this (with or without a port). */
adminHost?: string;
/**
* Stop the process when its main thread has been stuck this long (a function looping without
* awaiting), for a supervisor to restart it. Default 0: off. `onebase serve` turns it on at 30s.
*/
watchdogMs?: number;
/** Accept `onebase deploy` and run what it pushed, instead of the code given here (default true; never in dev). */
deploys?: boolean;
schema: SchemaDefinition<any> | null;
Expand Down Expand Up @@ -148,6 +154,8 @@ export async function startApp(opts: AppOptions) {
callLimits: opts.callLimits,
});
const { created, dropped } = await runtime.init();
const watchdog = opts.watchdogMs ? startWatchdog(opts.watchdogMs) : null;
runtime.watchdog = watchdog;
const server = serve({
runtime,
port: opts.port,
Expand Down Expand Up @@ -227,6 +235,7 @@ export async function startApp(opts: AppOptions) {
runtime.log.warn(`Shutting down with ${runtime.calls.active} call(s) still running after ${formatMs(graceMs)}.`);
}
runtime.stop();
watchdog?.stop();
await server.stop(true);
await driver.close();
};
Expand Down
2 changes: 2 additions & 0 deletions src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,8 @@ function serverOptions(opts: ReturnType<typeof options>) {
mutationMs: timeLimit("ONEBASE_MUTATION_TIMEOUT"),
actionMs: timeLimit("ONEBASE_ACTION_TIMEOUT"),
},
// Stops a server whose one thread is stuck, so its supervisor restarts it.
watchdogMs: timeLimit("ONEBASE_WATCHDOG") ?? 30_000,
callLimits: {
documentsRead: callLimit(CALL_LIMIT_ENV.documentsRead),
documentsWritten: callLimit(CALL_LIMIT_ENV.documentsWritten),
Expand Down
161 changes: 127 additions & 34 deletions src/engine/runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,11 @@ import { checkArgs, checkReturn, indexCode, lookupFunction, lookupRoute, type Mo
import { buildCatalog, TableScope, tableMount, validatesTable, type Catalog } from "./catalog";
import { CronRunner } from "./crons";
import { Scheduler } from "./scheduler";
import type { Watchdog } from "./watchdog";
import { BACKFILL_BATCH, backfilled, coalesce, hookState, readProgress, writeProgress, type HookProgress, type ResolvedHook } from "./hooks";
import { applySchema, DatabaseImpl, OnebaseError, readerView, writerView, type ReadSet, type VectorSearchQuery, type Write } from "./database";
import { isIdForTable, newId } from "./dialect";
import { envNumber, savepoint, Semaphore, SerialQueue, SqliteDriver, type Driver, type Exec } from "./driver";
import { envNumber, savepoint, Semaphore, SerialQueue, SqliteDriver, type Driver, type Exec, type Row } from "./driver";
import {
CONFIRMED,
DIRECT_UPLOAD_DEFAULT_SECONDS,
Expand Down Expand Up @@ -270,6 +271,13 @@ export type CallOptions = {
source?: CallSource;
};

/** Documents that don't match the schema: how many, and the first few. */
type InvalidDocuments = { invalid: number; examples: string[] };
/** A document check done ahead of a deploy's transaction: the schema it was for, and what was written during and after it. */
type PrecheckedDocuments = { fingerprint: string; touched: Set<string>; overflowed: boolean };
/** Past this many documents written during a precheck, the deploy checks everything again in its transaction. */
const MAX_PRECHECK_WRITES = 100_000;

/** What changed in a commit: exact writes, or (when too large to broadcast) just the tables. */
export type CommitEvent = ({ writes: Write[]; tables?: undefined } | { tables: string[]; writes?: undefined }) & { settle?: Settle };

Expand Down Expand Up @@ -396,6 +404,10 @@ export class Runtime {
/** Signs function handles; loaded from the database by init(). */
private handleKey: Buffer | null = null;
private stopped = false;
/** A deploy's document check in progress: every write adds the ids it touched (see precheckDocuments). */
private precheck: PrecheckedDocuments | null = null;
/** Set by the server: told which function starts, to name it if the thread gets stuck (watchdog.ts). */
watchdog: Watchdog | null = null;
private draining = false;
/** True once a shutdown has begun. */
get shuttingDown() {
Expand Down Expand Up @@ -528,52 +540,124 @@ export class Runtime {
return applied;
}

/**
* The tables whose documents are validated, and a fingerprint of their validators. With
* schemaValidation off only component tables are checked; turning it back on changes the
* fingerprint, so everything is checked again then.
*/
private checkedTables(catalog: Catalog) {
const tables = Object.entries(catalog.tables).filter(([t]) => validatesTable(catalog, t)) as [string, { validator: AnyValidator }][];
const fingerprint = new Bun.CryptoHasher("sha256").update(JSON.stringify(tables.map(([t, d]) => [t, d.validator]))).digest("hex");
return { tables, fingerprint };
}

private checkRows(table: string, def: { validator: AnyValidator }, rows: Row[], found: InvalidDocuments) {
for (const row of rows) {
const errors = validate(def.validator, this.driver.dialect.parseDoc(row.doc), "document");
if (!errors.length) continue;
found.invalid++;
if (found.examples.length < 5) found.examples.push(`${table} ${row.id}: ${errors.slice(0, 3).join("; ")}`);
}
}

/** Checks a table 500 documents at a time, read with `all`; `between` runs after each batch. */
private async checkTable(
table: string,
def: { validator: AnyValidator },
found: InvalidDocuments,
all: (sql: string, params: unknown[]) => Promise<Row[]>,
between?: () => Promise<void>,
) {
const d = this.driver.dialect;
let after: [number, string] = [-1, ""];
for (;;) {
const rows = await all(
`SELECT id, ct, doc FROM documents WHERE tbl = ? AND (ct, id) > (${d.placeholder("ct")}, ${d.placeholder("id")}) ORDER BY ct, id LIMIT 500`,
[table, ...after],
);
this.checkRows(table, def, rows, found);
if (rows.length < 500) break;
after = [Number(rows[rows.length - 1]!.ct), rows[rows.length - 1]!.id as string];
await between?.();
}
}

private schemaMismatch({ invalid, examples }: InvalidDocuments) {
return new OnebaseError(
`schema.ts doesn't match ${invalid} existing document${invalid === 1 ? "" : "s"}:\n - ${examples.join("\n - ")}` +
(invalid > examples.length ? `\n - … and ${invalid - examples.length} more` : "") +
`\nMake the schema accept the data (v.optional(...), v.union(...)), or fix the data: start with ` +
`defineSchema({ ... }, { schemaValidation: false }), update the documents in a mutation (or the dashboard), then turn validation back on.`,
);
}

/**
* A deploy's document check, without the write lock. Checking every document inside the
* deploy's transaction would hold up every write (and, with one thread, every request) for as
* long as the scan takes. So the scan happens first, a batch at a time with requests served in
* between, while the ids of documents written meanwhile are collected; the transaction then
* only re-checks those (checkDocuments).
*
* Returns null when there is nothing to gain: no schema, an unchanged one, or a shared database,
* where other instances write too and their writes can't be collected in time. The caller must
* call endPrecheck() once its transaction is over.
*/
private async precheckDocuments(catalog: Catalog | null): Promise<PrecheckedDocuments | null> {
if (!catalog || this.driver.shared) return null;
const { tables, fingerprint } = this.checkedTables(catalog);
const stored = await this.driver.read((tx) => tx.get("SELECT value FROM _meta WHERE \"key\" = 'schema'"));
if (stored?.value === fingerprint) return null;
const prechecked: PrecheckedDocuments = { fingerprint, touched: new Set(), overflowed: false };
this.precheck = prechecked;
const found: InvalidDocuments = { invalid: 0, examples: [] };
const read = (sql: string, params: unknown[]) => this.driver.read((tx) => tx.all(sql, params));
const breathe = () => new Promise<void>((resolve) => setImmediate(resolve));
try {
for (const [table, def] of tables) await this.checkTable(table, def, found, read, breathe);
} catch (err) {
this.endPrecheck();
throw err;
}
if (found.invalid) {
this.endPrecheck();
throw this.schemaMismatch(found);
}
return prechecked;
}

private endPrecheck() {
this.precheck = null;
}

/**
* When schema.ts (or a component's schema) changes, every stored document is checked against
* it, like a Convex deploy. The schema's fingerprint is saved once the data matches (unless
* `record` is false, for a dry run), so unchanged schemas cost nothing. Returns warnings about
* data the schema ignores.
*/
private async checkDocuments(tx: Exec, catalog: Catalog | null, record = true): Promise<string[]> {
private async checkDocuments(tx: Exec, catalog: Catalog | null, record = true, prechecked: PrecheckedDocuments | null = null): Promise<string[]> {
if (!catalog) {
if (record) await tx.run("DELETE FROM _meta WHERE \"key\" = 'schema'");
return [];
}
// With schemaValidation off only component tables are checked; turning it back on changes
// the fingerprint, so everything is checked again then.
const tables = Object.entries(catalog.tables).filter(([t]) => validatesTable(catalog, t)) as [string, { validator: AnyValidator }][];
const fingerprint = new Bun.CryptoHasher("sha256").update(JSON.stringify(tables.map(([t, d]) => [t, d.validator]))).digest("hex");
const { tables, fingerprint } = this.checkedTables(catalog);
const stored = await tx.get("SELECT value FROM _meta WHERE \"key\" = 'schema'");
if (stored?.value === fingerprint) return [];

const d = this.driver.dialect;
const examples: string[] = [];
let invalid = 0;
for (const [table, def] of tables) {
let after: [number, string] = [-1, ""];
for (;;) {
const rows = await tx.all(
`SELECT id, ct, doc FROM documents WHERE tbl = ? AND (ct, id) > (${d.placeholder("ct")}, ${d.placeholder("id")}) ORDER BY ct, id LIMIT 500`,
[table, ...after],
);
for (const row of rows) {
const errors = validate(def.validator, d.parseDoc(row.doc), "document");
if (!errors.length) continue;
invalid++;
if (examples.length < 5) examples.push(`${table} ${row.id}: ${errors.slice(0, 3).join("; ")}`);
}
if (rows.length < 500) break;
after = [Number(rows[rows.length - 1]!.ct), rows[rows.length - 1]!.id];
const found: InvalidDocuments = { invalid: 0, examples: [] };
if (prechecked?.fingerprint === fingerprint && !prechecked.overflowed) {
// Every document was checked a moment ago without the write lock (precheckDocuments). Only
// the ones written since then need another look, which is what keeps this transaction short.
const byTable = new Map(tables);
for (const id of prechecked.touched) {
const row = await tx.get("SELECT id, tbl, doc FROM documents WHERE id = ?", [id]);
const def = row ? byTable.get(row.tbl as string) : undefined;
if (def) this.checkRows(row!.tbl as string, def, [row!], found);
}
} else {
for (const [table, def] of tables) await this.checkTable(table, def, found, (sql, params) => tx.all(sql, params));
}
if (invalid) {
throw new OnebaseError(
`schema.ts doesn't match ${invalid} existing document${invalid === 1 ? "" : "s"}:\n - ${examples.join("\n - ")}` +
(invalid > examples.length ? `\n - … and ${invalid - examples.length} more` : "") +
`\nMake the schema accept the data (v.optional(...), v.union(...)), or fix the data: start with ` +
`defineSchema({ ... }, { schemaValidation: false }), update the documents in a mutation (or the dashboard), then turn validation back on.`,
);
}
if (found.invalid) throw this.schemaMismatch(found);
const known = new Set(Object.keys(catalog.tables));
const orphans = (await tx.all("SELECT DISTINCT tbl FROM documents"))
.map((r) => r.tbl as string)
Expand Down Expand Up @@ -895,8 +979,10 @@ export class Runtime {
? `The onebase CLI that pushed this didn't send its version, so it's older than this server (${VERSION}). Update it, so a version mismatch can be caught.`
: versionWarning(builtWith);
const schema = opts.dryRun ? this.dryRunSchema(opts.schema) : prepared!.catalog;
// The long part of a schema change, done before the transaction (and, for a dry run, instead of a long read).
const prechecked = await this.precheckDocuments(schema);
const apply = async (tx: Exec) => {
const warnings = await this.checkDocuments(tx, schema, !opts.dryRun);
const warnings = await this.checkDocuments(tx, schema, !opts.dryRun, prechecked);
if (opts.dryRun) return { indexes: { created: [], dropped: [] }, warnings };
const indexes = await applySchema(tx, this.driver.dialect, schema);
if (opts.save) await saveDeployment(tx, stored);
Expand All @@ -905,7 +991,7 @@ export class Runtime {
return { indexes, warnings };
};
// Queued with this process's mutations, so none of them runs between the check and the swap.
const { indexes, warnings } = opts.dryRun ? await this.driver.read(apply) : await this.withWriter(apply);
const { indexes, warnings } = await (opts.dryRun ? this.driver.read(apply) : this.withWriter(apply)).finally(() => this.endPrecheck());
const deployment = toDeployment(stored);
if (!opts.dryRun) {
await this.install(prepared!, deployment);
Expand Down Expand Up @@ -1705,6 +1791,7 @@ export class Runtime {
const checked = this.checkArgs(name, fn, args);
const caller = this.componentFunctions.get(name)?.mount ?? null;
const handle = (ref: unknown) => this.createHandle(ref, caller);
this.watchdog?.enter(name);
const result = await currentFunction.run({ type: fn.type, name, handle, reads }, () => fn.handler(ctx, checked));
return this.checkReturn(name, fn, result);
}
Expand Down Expand Up @@ -1867,6 +1954,12 @@ export class Runtime {
// Nothing commits before every table hook has seen every write.
await this.flushHooks(state);
budget?.check(); // a limit error the function caught still rolls the mutation back
// Before the commit, in the writer: a deploy's transaction, queued behind this one, sees every id.
const precheck = this.precheck;
if (precheck && state.writes.length) {
if (precheck.touched.size + state.writes.length > MAX_PRECHECK_WRITES) precheck.overflowed = true;
else for (const w of state.writes) precheck.touched.add(w.id);
}
if (this.driver.shared && this.driver.publish && (state.writes.length || state.scheduled || state.revoked.length)) {
for (const message of this.broadcastMessages(state)) await this.driver.publish(tx, message);
}
Expand Down
Loading
Loading