Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions bun.lock

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/backups.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ onebase serve --backup s3://my-bucket/backups --backup-every 1h --backup-keep 24
- **Snapshots:** `VACUUM INTO` gives a consistent copy without blocking writes. It's written next to the database (so allow free space for one more copy), checked with SQLite's `quick_check`, gzipped and uploaded as `db/onebase-<time>.sqlite.gz`. A snapshot that fails the check isn't uploaded, and the failure is logged.
- **Schedule:** the first backup runs a few seconds after startup, then one every `--backup-every` (`30m`, `1h`, `1d`, …).
- **Retention:** the newest `--backup-keep` snapshots are kept (default 24), and also the newest snapshot of each of the last `--backup-keep-daily` days (default 7). With hourly backups that is a day of hourly copies and a week of daily ones, for the mistake you only notice days later. `--backup-keep-daily 0` keeps just the newest ones.
- **Encryption:** with `ONEBASE_SECRET_KEY` set (at least 32 characters; `openssl rand -base64 32`), snapshots are encrypted before upload (AES-256-GCM) and named `….sqlite.gz.enc`. Restoring one needs the same key, so keep it somewhere other than the bucket. Uploaded files are copied as they are. Without the key, treat the bucket like the database itself: a snapshot holds your data, password hashes and the code you deployed, and whoever can replace a snapshot decides what a restore runs.
- **Encryption:** with `ONEBASE_SECRET_KEY` set (at least 32 characters; `openssl rand -base64 32`), snapshots are encrypted before upload (AES-256-GCM) and named `….sqlite.gz.enc`. Restoring one needs the same key, so keep it somewhere other than the bucket. Uploaded files are encrypted the same way (as `files/<id>.enc`), and files that were backed up before the key was set are replaced with encrypted copies on the next backup. Without the key, treat the bucket like the database itself: a snapshot holds your data, password hashes and the code you deployed, and whoever can replace a snapshot decides what a restore runs.
- **Files:** uploaded files in the local `files/` folder are synced to `files/` in the bucket. Only new files are uploaded, since files never change.

How much data you can lose depends on the interval: with `--backup-every 15m`, at most 15 minutes of writes.
Expand All @@ -44,6 +44,6 @@ onebase restore --backup s3://my-bucket/backups --data ./onebase_data
onebase restore --backup s3://my-bucket/backups --at 20261022T0300 --force
```

`restore` downloads the newest snapshot, or the one whose name contains `--at`, into `<data>/db.sqlite`, and downloads any files that are missing. It refuses to overwrite an existing database unless you pass `--force`. The download is checked before it replaces anything: a damaged or wrongly decrypted snapshot is an error, and the database that was there stays. For an encrypted snapshot, set `ONEBASE_SECRET_KEY` to the key the server had.
`restore` downloads the newest snapshot, or the one whose name contains `--at`, into `<data>/db.sqlite`, and downloads any files that are missing. It refuses to overwrite an existing database unless you pass `--force`. The download is checked before it replaces anything: a damaged or wrongly decrypted snapshot is an error, and the database that was there stays. For an encrypted snapshot or encrypted files, set `ONEBASE_SECRET_KEY` to the key the server had.

You can also use it to move a project to a new server: install onebase there, run `restore`, then `serve`.
1 change: 1 addition & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ Each flag can also be set with an environment variable, which can come from a [.
| `--no-dashboard` | dashboard on / `$ONEBASE_DASHBOARD=off` | Don't serve the dashboard or admin API at `/_admin` (this also turns off deploys) |
| `--no-deploy` | deploys on / `$ONEBASE_DEPLOY=off` | Refuse `onebase deploy` and run the local code instead of what was pushed |
| `--check` | | `upgrade`: only say whether a newer release exists |
| `--no-verify` | | `upgrade`: don't check the release's signed build attestation |
| `--dry-run` | | `deploy`: check the code against the server's data without changing anything |
| `--rollback` | | `deploy`: go back to the previous deployment |
| `--env-file <path>` | `.env` files for the command | Read only this file; repeat for several |
Expand Down
2 changes: 1 addition & 1 deletion docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ onebase upgrade # or: onebase upgrade 0.3.0
systemctl restart onebase # or however you run it
```

`onebase upgrade` downloads the release for your platform from [GitHub](https://github.com/abdullah-bl/onebase/releases), checks it against the release's `SHA256SUMS`, makes sure it runs, and only then swaps it in. The old binary stays next to it as `onebase.old`. The running server keeps the old version until it restarts, and clients reconnect by themselves after a restart, replaying any mutations they queued meanwhile.
`onebase upgrade` downloads the release for your platform from [GitHub](https://github.com/abdullah-bl/onebase/releases), checks it against the release's `SHA256SUMS`, checks its signed build attestation, makes sure it runs, and only then swaps it in. The attestation is what says who built the binary: it proves that GitHub Actions built this exact file from this repository's release workflow at the release's tag, which a checksum stored next to the binary can't. The check needs to reach GitHub's API and Sigstore; on a machine that can't, check the download elsewhere with `gh attestation verify` and pass `--no-verify`. Releases before 0.3.0 weren't signed, so only their checksum is checked. The old binary stays next to it as `onebase.old`. The running server keeps the old version until it restarts, and clients reconnect by themselves after a restart, replaying any mutations they queued meanwhile.

- **Docker:** pull the newer image and recreate the container: `docker compose pull && docker compose up -d`. The deployment and data live on the volume.
- **Several instances** (Postgres, MySQL or Turso): restart them one at a time, so some instance always serves. Don't run two processes on one SQLite file: restarting is a few seconds of downtime there.
Expand Down
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,8 @@
"@types/react": "19",
"preact": "10",
"react": "19",
"shiki": "^4.5.0"
"shiki": "^4.5.0",
"sigstore": "^5.0.1"
},
"peerDependencies": {
"typescript": "^7",
Expand Down
6 changes: 4 additions & 2 deletions src/auth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -497,9 +497,11 @@ export function apple(
throw new Error("Apple sign-in needs the random nonce it started with (at least 16 characters).");
}
const { claims, profile } = await readToken(idToken, audiences);
// Apple echoes the nonce the app gave it: normally sha256(rawNonce) in hex, or the raw nonce itself.
// The app gives Apple sha256(rawNonce) in hex, Apple puts that in the token, and the app sends
// the raw nonce here. Only the hash is accepted: if the token's own nonce claim were too, anyone
// holding a token could read the claim out of it and send that, and the check would prove nothing.
const echoed = typeof claims.nonce === "string" ? claims.nonce : "";
if (!echoed || (echoed !== (await sha256Hex(nonce)) && echoed !== nonce)) throw new Error("This Apple sign-in doesn't match the one that was started.");
if (!echoed || echoed !== (await sha256Hex(nonce))) throw new Error("This Apple sign-in doesn't match the one that was started.");
const exp = typeof claims.exp === "number" ? claims.exp * 1000 : Date.now() + 10 * 60_000;
return { ...profile, name: cleanName(params.name), once: { key: `apple|${claims.sub}|${echoed}|${exp}`, until: exp + 5 * 60_000 } };
},
Expand Down
43 changes: 33 additions & 10 deletions src/backup.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ export const DEFAULT_KEEP_DAILY = 7;
export type BackupResult = { key: string; bytes: number; filesUploaded: number };

const SNAPSHOT_PREFIX = "db/onebase-";
/** Storage ids are lowercase letters and digits. */
const STORAGE_ID = /^[0-9a-z]+$/;

function stamp(date = new Date()): string {
return date.toISOString().replace(/[-:]/g, "").replace(/\.\d+Z$/, "Z");
Expand Down Expand Up @@ -87,15 +89,21 @@ export async function backupNow(driver: Driver, url: string, keep: number, blobs
const snapshots = (await listKeys(loc, SNAPSHOT_PREFIX)).map((o) => o.key);
for (const old of snapshotsToDelete(snapshots, keep, keepDaily)) await loc.client.delete(old);

// Files never change once written, so only new ones are uploaded.
// Files never change once written, so only new ones are uploaded. With a secret key they are
// encrypted like the snapshot (as <id>.enc), and a copy uploaded in the clear before the key was
// set is replaced: otherwise "my backups are encrypted" would be true of new files only.
let filesUploaded = 0;
if (blobs instanceof LocalBlobs && existsSync(blobs.dir)) {
const remote = new Set((await listKeys(loc, "files/")).map((o) => o.key));
for (const name of readdirSync(blobs.dir)) {
const key = `${loc.prefix}files/${name}`;
if (remote.has(key)) continue;
await loc.client.write(key, Bun.file(join(blobs.dir, name)));
filesUploaded++;
const plain = `${loc.prefix}files/${name}`;
const key = secret ? `${plain}.enc` : plain;
if (!remote.has(key)) {
const file = Bun.file(join(blobs.dir, name));
await loc.client.write(key, secret ? new Response(file.stream().pipeThrough(encryptStream(secret))) : file);
filesUploaded++;
}
if (secret && remote.has(plain)) await loc.client.delete(plain);
}
}
return { key, bytes, filesUploaded };
Expand Down Expand Up @@ -173,11 +181,26 @@ export async function restore(url: string, dbPath: string, filesDir: string, opt
let files = 0;
mkdirSync(filesDir, { recursive: true });
for (const { key } of await listKeys(loc, "files/")) {
const name = key.slice(`${loc.prefix}files/`.length);
// Storage ids are lowercase letters and digits. Any other key isn't one of ours, and a name
// like "../../x" would be written outside the files folder.
if (!/^[0-9a-z]+$/.test(name) || existsSync(join(filesDir, name))) continue;
await Bun.write(join(filesDir, name), loc.client.file(key));
const stored = key.slice(`${loc.prefix}files/`.length);
const sealed = stored.endsWith(".enc");
const name = sealed ? stored.slice(0, -".enc".length) : stored;
// Any key that isn't a storage id isn't one of ours, and a name like "../../x" would be
// written outside the files folder.
if (!STORAGE_ID.test(name) || existsSync(join(filesDir, name))) continue;
if (!sealed) {
await Bun.write(join(filesDir, name), loc.client.file(key));
} else {
const secret = secretKey();
if (!secret) throw new Error(`The file ${key} is encrypted. Set $${SECRET_KEY_ENV} to the key the server had when it was backed up.`);
// Under another name until it's whole: a file that fails to decrypt half way isn't left looking restored.
const tmp = join(filesDir, `.restore-${newId()}`);
try {
await Bun.write(tmp, new Response(loc.client.file(key).stream().pipeThrough(decryptStream(secret))));
renameSync(tmp, join(filesDir, name));
} finally {
rmSync(tmp, { force: true });
}
}
files++;
}
return { key: chosen.key, files };
Expand Down
4 changes: 3 additions & 1 deletion src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ Options (each $NAME can also come from a .env file):
--no-deploy Refuse onebase deploy and run the local code, not what was pushed ($ONEBASE_DEPLOY=off)
--from-file <f> env set: set every variable in this .env file
--check upgrade: only say whether a newer release exists
--no-verify upgrade: don't check the release's signed build attestation
--dry-run deploy: check the code against the server's data without changing anything
--rollback deploy: go back to the previous deployment
--env-file <path> Read this .env file instead of the defaults (repeatable)
Expand Down Expand Up @@ -146,6 +147,7 @@ function options(argv: string[]) {
"no-deploy": { type: "boolean" },
"dry-run": { type: "boolean", default: false },
check: { type: "boolean", default: false },
"no-verify": { type: "boolean", default: false },
rollback: { type: "boolean", default: false },
"env-file": { type: "string", multiple: true },
"from-file": { type: "string" },
Expand Down Expand Up @@ -534,7 +536,7 @@ export async function main(argv = process.argv.slice(2)) {
case "env":
return envCommand({ args: opts.args, url: opts.url, key: process.env.ONEBASE_DEPLOY_KEY?.trim(), fromFile: opts["from-file"] });
case "upgrade":
await upgrade({ version: opts.args[0], check: opts.check, compiled: COMPILED });
await upgrade({ version: opts.args[0], check: opts.check, compiled: COMPILED, verify: !opts["no-verify"] });
return;
case "build":
return buildCommand(opts);
Expand Down
8 changes: 8 additions & 0 deletions src/engine/driver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -673,6 +673,14 @@ export class LibsqlDriver implements Driver {
return result;
} catch (err) {
await tx.rollback().catch(() => {});
if ((err as any)?.code === "TRANSACTION_TIMEOUT") {
// Not a LibsqlError, so it isn't retried: the same work would run out of time again.
throw new Error(
"The libSQL server ended this transaction because it stayed open too long (sqld and Turso allow about 5 seconds). " +
"Every read and write in a function is a round trip to the server, so do less in one mutation: write in several smaller ones, or schedule the rest.",
{ cause: err },
);
}
throw err;
} finally {
tx.close();
Expand Down
62 changes: 62 additions & 0 deletions src/upgrade.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,54 @@ export type UpgradeOptions = {
current?: string;
fetch?: (url: string, init?: RequestInit) => Promise<Response>;
log?: (line: string) => void;
/** Set to false to skip checking who built the binary (--no-verify). */
verify?: boolean;
/** Checks who built the binary; throws if it can't tell (default: verifyProvenance). */
provenance?: (binary: Provenance) => Promise<void>;
};

/** A downloaded release binary: its file name, SHA-256 (hex) and the release's tag. */
export type Provenance = { name: string; digest: string; tag: string; fetch: NonNullable<UpgradeOptions["fetch"]> };

/** The first release whose binaries were signed. Older ones can only be checked against SHA256SUMS. */
const FIRST_ATTESTED = "0.3.0";

/**
* Checks that GitHub Actions built this exact binary from this repository's release workflow, at
* the release's tag. SHA256SUMS can't say that: it sits in the same release, so whoever could
* replace the binary could replace it too. The attestation is signed with a short-lived
* certificate that Sigstore issued to that workflow run and recorded in a public log; the
* signature, the certificate's chain, the log entry and the workflow it names are all checked.
*/
export async function verifyProvenance({ name, digest, tag, fetch: get }: Provenance): Promise<void> {
const api = `https://api.github.com/repos/${REPO}/attestations/sha256:${digest}`;
const res = await get(api, { headers: { Accept: "application/vnd.github+json", "User-Agent": "onebase-cli" } });
if (!res.ok && res.status !== 404) throw new Error(`GitHub answered ${res.status} for ${api}${res.status === 403 ? " (rate limited? try again in a while)" : ""}.`);
const attestations = res.ok ? (((await res.json()) as { attestations?: { bundle: any }[] }).attestations ?? []) : [];
if (!attestations.length) throw new Error(`GitHub has no build attestation for ${name} (sha256 ${digest}), so there's no telling who built it.`);

const { verify } = await import("sigstore");
let reason: unknown;
for (const { bundle } of attestations) {
try {
await verify(bundle, {
certificateIssuer: "https://token.actions.githubusercontent.com",
certificateIdentityURI: `https://github.com/${REPO}/.github/workflows/release.yml@refs/tags/${tag}`,
});
// The signature covers this statement, and the statement names the files it is about.
const statement = JSON.parse(Buffer.from(bundle.dsseEnvelope.payload, "base64").toString());
if (bundle.dsseEnvelope.payloadType !== "application/vnd.in-toto+json" || statement.predicateType !== "https://slsa.dev/provenance/v1") {
throw new Error("it isn't a build provenance statement");
}
if (!statement.subject?.some((s: any) => s?.name === name && s?.digest?.sha256 === digest)) throw new Error("it is about a different file");
return;
} catch (err) {
reason = err;
}
}
throw new Error(`The build attestation for ${name} doesn't hold: ${reason instanceof Error ? reason.message : String(reason)}.`);
}

export type UpgradeResult = { status: "up-to-date" | "available" | "upgraded" | "not-a-binary"; current: string; release?: string };

type Release = { tag_name: string; assets: { name: string; browser_download_url: string }[] };
Expand Down Expand Up @@ -91,6 +137,22 @@ export async function upgrade(opts: UpgradeOptions): Promise<UpgradeResult> {
const actual = new Bun.CryptoHasher("sha256").update(binary).digest("hex");
if (actual !== expected.toLowerCase()) throw new Error(`${name} doesn't match its SHA-256 in SHA256SUMS (got ${actual}). Nothing was changed.`);

if (opts.verify === false) {
log(` Not checking who built it (--no-verify).`);
} else if (compareVersions(target, FIRST_ATTESTED) < 0) {
log(` onebase ${target} is from before releases were signed, so only its checksum was checked.`);
} else {
try {
await (opts.provenance ?? verifyProvenance)({ name, digest: actual, tag: release.tag_name, fetch: get });
} catch (err) {
throw new Error(
`${err instanceof Error ? err.message : String(err)}\nNothing was changed. If this machine can't reach Sigstore or GitHub's API, ` +
`check the binary elsewhere with gh attestation verify ${name} --repo ${REPO}, then pass --no-verify.`,
);
}
log(` Built by this repository's release workflow at ${release.tag_name} (signed attestation checked).`);
}

const exec = opts.execPath ?? process.execPath;
const fresh = `${exec}.new`;
const old = `${exec}.old`;
Expand Down
8 changes: 6 additions & 2 deletions tests/auth-apple-email.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -235,10 +235,14 @@ describe("Sign in with Apple and email codes", () => {
expect((await appleSignIn({ idToken: bare.idToken, nonce: bare.nonce })).status).toBe(401);
});

test("accepts the raw nonce echoed back, as web sign-in does", async () => {
test("refuses the token's own nonce claim sent back as the nonce", async () => {
// What someone holding only the token can do: read the claim out of it. Hashed or not, it isn't the raw nonce.
const hashed = await fake.identity({ sub: "apple-claim" });
const claim = JSON.parse(Buffer.from(hashed.idToken.split(".")[1]!, "base64url").toString()).nonce;
expect((await appleSignIn({ idToken: hashed.idToken, nonce: claim })).status).toBe(401);
const nonce = b64(crypto.getRandomValues(new Uint8Array(16)));
const { idToken } = await fake.identity({ sub: "apple-raw", nonce });
expect((await appleSignIn({ idToken, nonce })).status).toBe(200);
expect((await appleSignIn({ idToken, nonce })).status).toBe(401);
});

test("refuses tokens for another app, another issuer, an expired token and a forged signature", async () => {
Expand Down
Loading
Loading