diff --git a/.claude/skills/doc-check/SKILL.md b/.claude/skills/doc-check/SKILL.md index 4d0fce32..39a453de 100644 --- a/.claude/skills/doc-check/SKILL.md +++ b/.claude/skills/doc-check/SKILL.md @@ -92,6 +92,7 @@ Code examples embedded in prose. API changes silently break copy-paste. | File | Why it needs checking | | --------------------------------- | ------------------------------------- | | `website/guide/quick-start.md` | First code users copy-paste | +| `website/guide/databases.md` | Database comparison and setup codes | | `website/guide/concepts.md` | Core concept explanations with code | | `website/guide/server-mode.md` | Server mode tutorial | | `website/guide/fullstack-mode.md` | Fullstack mode tutorial | @@ -101,11 +102,11 @@ Code examples embedded in prose. API changes silently break copy-paste. | `website/guide/multi-tenant.md` | Multi-tenant isolation patterns | | `website/guide/deployment.md` | Deployment guide with mode comparison | -### Tier 6: Sidebar config +### Tier 6: Sidebar config and meta tags -Menu links and anchors must match actual headings. Mismatches cause 404s. +Menu links, anchors, and OGP/meta descriptions must stay in sync. -- [ ] `website/.vitepress/config.ts` — VitePress slugifies headings for anchors +- [ ] `website/.vitepress/config.ts` — sidebar links, `og:description`, `twitter:description`, site `description` ### Tier 7: Example apps diff --git a/.claude/skills/doc-check/scripts/find-stale.sh b/.claude/skills/doc-check/scripts/find-stale.sh index ecf85306..7d2a8fcb 100755 --- a/.claude/skills/doc-check/scripts/find-stale.sh +++ b/.claude/skills/doc-check/scripts/find-stale.sh @@ -133,6 +133,18 @@ else "Prefer await durably.init() in guides and examples" \ "$API_REF_EXCLUDE" + # ── OGP / meta tags ── + + check_pattern \ + "OGP/meta still says 'Just SQLite'" \ + 'Just SQLite' \ + "PostgreSQL is now supported. Update og:description, twitter:description, and site description" + + check_pattern \ + "OGP/meta still says 'No Redis'" \ + 'No Redis' \ + "Outdated positioning. Update og:description and twitter:description" + # ── Old terminology ── check_pattern \ diff --git a/README.md b/README.md index b27f7f39..bfb7026a 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # durably -Step-oriented resumable batch execution for Node.js and browsers using SQLite. +Steps that survive crashes. SQLite to PostgreSQL. **[Documentation](https://coji.github.io/durably/)** | **[Live Demo](https://durably-demo.vercel.app)** @@ -13,16 +13,21 @@ Step-oriented resumable batch execution for Node.js and browsers using SQLite. ## Features -- Resumable batch processing with step-level persistence -- Works in both Node.js and browsers -- Uses SQLite for state management (better-sqlite3/libsql for Node.js, SQLite WASM for browsers) -- Minimal dependencies - just Kysely and Zod as peer dependencies -- Event system for monitoring and extensibility -- Type-safe input/output with Zod schemas +- **Resumable** — each step's result is persisted; interrupted jobs resume from the last successful step +- **Flexible storage** — libSQL/Turso, PostgreSQL, better-sqlite3, or browser OPFS +- **Browser + server** — same API for Node.js and browsers +- **Lease-based recovery** — stale workers are automatically reclaimed via fencing tokens +- **Auto cleanup** — `retainRuns` option purges old completed runs automatically +- **React hooks** — real-time progress via SSE, fullstack and SPA modes +- **Type-safe** — Zod schemas for input/output, labels, and auth context ## Quick Start -See the [Quick Start](https://coji.github.io/durably/guide/quick-start) for installation and usage instructions. +```bash +pnpm add @coji/durably kysely zod @libsql/client @libsql/kysely-libsql +``` + +See the [Quick Start](https://coji.github.io/durably/guide/quick-start) guide, or [Choosing a Database](https://coji.github.io/durably/guide/databases) for PostgreSQL and other backends. ## License diff --git a/packages/durably-react/README.md b/packages/durably-react/README.md index d2ec0163..9539f4c4 100644 --- a/packages/durably-react/README.md +++ b/packages/durably-react/README.md @@ -1,6 +1,6 @@ # @coji/durably-react -React bindings for [Durably](https://github.com/coji/durably) - step-oriented resumable batch execution. +React bindings for [Durably](https://github.com/coji/durably) — steps that survive crashes. **[Documentation](https://coji.github.io/durably/)** | **[GitHub](https://github.com/coji/durably)** diff --git a/packages/durably/README.md b/packages/durably/README.md index 2baa81be..ca594f1a 100644 --- a/packages/durably/README.md +++ b/packages/durably/README.md @@ -1,6 +1,6 @@ # @coji/durably -Step-oriented resumable batch execution for Node.js and browsers using SQLite. +Steps that survive crashes. SQLite to PostgreSQL. **[Documentation](https://coji.github.io/durably/)** | **[GitHub](https://github.com/coji/durably)** | **[Live Demo](https://durably-demo.vercel.app)** @@ -9,10 +9,14 @@ Step-oriented resumable batch execution for Node.js and browsers using SQLite. ## Installation ```bash -npm install @coji/durably kysely zod better-sqlite3 +# libSQL (recommended default) +npm install @coji/durably kysely zod @libsql/client @libsql/kysely-libsql + +# PostgreSQL (multi-worker) +npm install @coji/durably kysely zod pg ``` -See the [Quick Start](https://coji.github.io/durably/guide/quick-start) for other SQLite backends (libsql, SQLocal for browsers). +See [Choosing a Database](https://coji.github.io/durably/guide/databases) for all backends. ## Quick Start diff --git a/packages/durably/docs/llms.md b/packages/durably/docs/llms.md index 523520ad..527b8240 100644 --- a/packages/durably/docs/llms.md +++ b/packages/durably/docs/llms.md @@ -4,7 +4,7 @@ ## Overview -Durably is a minimal workflow engine that persists step results to SQLite or PostgreSQL. If a job is interrupted (server restart, browser tab close, crash), it automatically resumes from the last successful step. Supports libSQL/Turso (single-server, serverless), PostgreSQL (multi-worker), and SQLocal (browser/OPFS). +Durably is a minimal workflow engine that persists step results to SQLite or PostgreSQL. If a job is interrupted (server restart, browser tab close, crash), it automatically resumes from the last successful step. Supports libSQL/Turso (single-server or serverless), PostgreSQL (recommended for multi-worker), and SQLocal (browser/OPFS). ## Installation diff --git a/website/.vitepress/config.ts b/website/.vitepress/config.ts index 2984ee58..e85dcce9 100644 --- a/website/.vitepress/config.ts +++ b/website/.vitepress/config.ts @@ -3,7 +3,7 @@ import { defineConfig } from 'vitepress' export default defineConfig({ title: 'Durably', description: - 'Step-oriented resumable batch execution for Node.js and browsers', + 'Steps that survive crashes. Resumable batch execution for Node.js and browsers. SQLite to PostgreSQL.', base: '/durably/', head: [ @@ -13,7 +13,7 @@ export default defineConfig({ 'meta', { property: 'og:description', - content: 'Just SQLite. No Redis required.', + content: 'Steps that survive crashes. SQLite to PostgreSQL.', }, ], [ @@ -30,7 +30,7 @@ export default defineConfig({ 'meta', { name: 'twitter:description', - content: 'Just SQLite. No Redis required.', + content: 'Steps that survive crashes. SQLite to PostgreSQL.', }, ], [ diff --git a/website/guide/databases.md b/website/guide/databases.md index 985f6459..0a636319 100644 --- a/website/guide/databases.md +++ b/website/guide/databases.md @@ -4,13 +4,24 @@ Durably supports multiple database backends through Kysely dialects. This guide ## At a Glance -| Backend | Writers | Serverless | Setup | Performance | Cost | -| ------------------------- | ------------- | ---------- | --------------- | ------------------------------ | ----------------------------- | -| **libSQL** (local) | Single | Limited | Zero config | Fast (local file) | Free | -| **Turso** (remote libSQL) | Single per DB | Yes | Managed | Network latency, edge-friendly | Free tier, then pay-as-you-go | -| **better-sqlite3** | Single | No | Zero config | Fast (local sync) | Free | -| **PostgreSQL** | Multiple | Varies | Server required | Strong under concurrency | Self-hosted or managed | -| **SQLocal** (browser) | Single tab | N/A | Zero config | Browser-local (OPFS) | Free (client-side) | +### SQLite Family (Single-Writer) + +| Backend | Serverless | Setup | Performance | Cost | +| --------------------- | --------------------------- | -------------------------------------- | -------------------------------- | ------------------------------------------------ | +| **libSQL / Turso** | Local: Limited, Remote: Yes | Zero config (local) / Managed (remote) | Fast local, edge-friendly remote | Free (local) / Free tier + pay-as-you-go (Turso) | +| **better-sqlite3** | No | Zero config | Fast (synchronous) | Free | +| **SQLocal** (browser) | N/A | Zero config | Browser-local (OPFS) | Free (client-side) | + +### PostgreSQL (Multi-Writer) + +- **Serverless**: Varies (Neon: yes, RDS: no) +- **Setup**: Server required +- **Performance**: Strong under concurrency (advisory locks + `FOR UPDATE SKIP LOCKED`) +- **Cost**: Self-hosted or managed (Neon, Supabase, RDS) + +::: warning +**Local SQLite** (libSQL local, better-sqlite3) is single-writer — multiple workers on the same file will cause lock contention. **Turso remote** accepts multiple connections, but concurrency key enforcement is weaker than PostgreSQL (no advisory locks). For reliable multi-worker setups, use PostgreSQL. +::: ## Decision Flowchart @@ -19,21 +30,23 @@ Running in the browser? Yes → SQLocal (only option) No ↓ -Need multiple workers processing jobs concurrently? - Yes → PostgreSQL +Large volume of jobs, or multiple app servers sharing one DB? +(e.g. high-traffic API, queue with thousands of jobs/hour) + Yes → PostgreSQL (strongest guarantees) + Turso also works (weaker concurrency key enforcement) No ↓ Deploying to serverless / edge (Vercel, Cloudflare)? Yes → Turso (remote libSQL) No ↓ -Need a lightweight embedded DB? +Single server or CLI script? Yes → libSQL (local) or better-sqlite3 ``` -## libSQL (Local) +## libSQL / Turso -**Recommended default for Node.js.** Zero-config embedded database that works everywhere. +**Recommended default.** libSQL works as a local embedded database and as a managed remote database via [Turso](https://turso.tech). Same `LibsqlDialect` for both — just change the URL. ```bash pnpm add @libsql/client @libsql/kysely-libsql @@ -43,36 +56,21 @@ pnpm add @libsql/client @libsql/kysely-libsql import { createClient } from '@libsql/client' import { LibsqlDialect } from '@libsql/kysely-libsql' +// Local (single file on disk) const client = createClient({ url: 'file:local.db' }) -const dialect = new LibsqlDialect({ client }) -``` - -- Single file on disk -- No external dependencies -- Same dialect works with Turso remote (just change the URL) - -## Turso (Remote libSQL) - -**For serverless and edge deployments.** Managed libSQL database with global replication. -```bash -pnpm add @libsql/client @libsql/kysely-libsql -``` +// Turso remote (serverless / edge) +// const client = createClient({ +// url: process.env.TURSO_DATABASE_URL!, +// authToken: process.env.TURSO_AUTH_TOKEN!, +// }) -```ts -import { createClient } from '@libsql/client' -import { LibsqlDialect } from '@libsql/kysely-libsql' - -const client = createClient({ - url: process.env.TURSO_DATABASE_URL!, - authToken: process.env.TURSO_AUTH_TOKEN!, -}) const dialect = new LibsqlDialect({ client }) ``` -- Same `LibsqlDialect` as local — just swap the URL +- **Local**: Single file, zero config, no external dependencies +- **Turso**: Managed service with global replication, free tier available - Works on Vercel, Cloudflare Workers, Fly.io -- Free tier available at [turso.tech](https://turso.tech) ::: tip See the [fullstack-vercel-turso example](https://github.com/coji/durably/tree/main/examples/fullstack-vercel-turso) for a complete Vercel + Turso deployment. @@ -97,11 +95,11 @@ const dialect = new SqliteDialect({ - Synchronous API (slightly faster for small workloads) - Good for one-off scripts and CLI tools -- No Turso remote support (use libSQL if you might need remote later) +- No remote support (use libSQL if you might need Turso later) ## PostgreSQL -**For multi-worker production deployments.** The only backend that supports multiple workers processing jobs concurrently from the same database. +**For multi-worker production deployments.** The recommended backend for running multiple workers concurrently, with advisory locks and `FOR UPDATE SKIP LOCKED` for strong concurrency guarantees. ```bash pnpm add pg @@ -122,10 +120,6 @@ const dialect = new PostgresDialect({ - Connection pooling via `pg.Pool` - Works with any PostgreSQL provider (Neon, Supabase, RDS, self-hosted) -::: warning -SQLite backends (libSQL, better-sqlite3) are single-writer. Running multiple workers against the same SQLite file will cause lock contention. Use PostgreSQL for multi-worker setups. -::: - ## SQLocal (Browser) **For browser-only apps.** Runs SQLite in the browser using OPFS (Origin Private File System). diff --git a/website/guide/index.md b/website/guide/index.md index 5fcdf497..bb0b5289 100644 --- a/website/guide/index.md +++ b/website/guide/index.md @@ -11,7 +11,7 @@ Long-running tasks fail. Networks drop, servers restart, browsers close. Traditi ## The Solution -Durably saves each step's result to SQLite. On resume, completed steps return cached results instantly. +Durably saves each step's result to the database (SQLite or PostgreSQL). On resume, completed steps return cached results instantly. ```ts const job = defineJob({ @@ -35,14 +35,28 @@ If the process crashes after importing 500 of 1000 rows, restart picks up at row ## Three Ways to Run -| Mode | Storage | Use Case | -| ------------- | ------------------------------ | ------------------------------------ | -| **Server** | @libsql/client, better-sqlite3 | Cron jobs, data pipelines, CLI tools | -| **Fullstack** | Server DB + SSE to browser | Web apps with real-time progress UI | -| **SPA** | SQLite WASM + OPFS | Offline-capable, local-first apps | +| Mode | Storage | Use Case | +| ------------- | ---------------------------------- | ------------------------------------ | +| **Server** | libSQL, PostgreSQL, better-sqlite3 | Cron jobs, data pipelines, CLI tools | +| **Fullstack** | Server DB + SSE to browser | Web apps with real-time progress UI | +| **SPA** | SQLite WASM + OPFS | Offline-capable, local-first apps | Same job definition works in all three modes. +## When NOT to Use Durably + +Durably is great for batch processing with step-level resumability. It's not the right tool for everything: + +| If you need... | Use instead | +| -------------------------------------------------------------- | ------------------------------------------------------ | +| **Sub-second job dispatch** (real-time messaging, webhooks) | BullMQ, SQS, or a message broker | +| **Hundreds of concurrent workers** processing a massive queue | Temporal, Inngest, or a dedicated workflow engine | +| **Simple scheduled tasks** (one cron job, no steps) | Node.js cron (`node-cron`), OS crontab, or Vercel Cron | +| **Stream processing** (continuous event ingestion) | Kafka, Redis Streams | +| **Long-running single operation** (no natural step boundaries) | A plain async function with retry logic | + +Durably shines when your task has **multiple discrete steps** that are expensive to redo, and you want **automatic resume** without managing infrastructure. + ## Next Step **[Quick Start](/guide/quick-start)** — Run your first resumable job in under 2 minutes. diff --git a/website/index.md b/website/index.md index 263c5523..db5305b8 100644 --- a/website/index.md +++ b/website/index.md @@ -4,7 +4,7 @@ layout: home hero: name: Durably text: Resumable Batch Execution - tagline: Just SQLite. No Redis required. + tagline: Steps that survive crashes. SQLite to PostgreSQL. actions: - theme: brand text: Get Started @@ -17,16 +17,22 @@ hero: link: https://github.com/coji/durably features: - - icon: 🚀 - title: Zero Infrastructure - details: SQLite only. No Redis, no Postgres, no config files. Just npm install and go. - icon: 🔄 title: Resumable Steps - details: Each step auto-saves to SQLite. Interrupted jobs resume exactly where they left off. + details: Each step auto-saves to the database. Interrupted jobs resume exactly where they left off — server restarts, crashes, browser tab closes. + - icon: 🗄️ + title: Flexible Storage + details: SQLite, libSQL/Turso, PostgreSQL, or browser OPFS. Pick what fits your deployment — from zero-config local to multi-worker production. - icon: 🌐 title: Browser + Server - details: Same API for Node.js and browsers. Use OPFS for offline-capable browser apps. + details: Same API for Node.js and browsers. Use OPFS for offline-capable browser apps, or connect to a server via SSE. - icon: ⚡ title: React Ready - details: Built-in hooks with real-time updates via SSE. Build progress UIs in minutes. + details: Built-in hooks with real-time progress updates via SSE. Fullstack and SPA modes with type-safe APIs. + - icon: 🔒 + title: Lease-Based Recovery + details: Workers claim jobs via leases with fencing tokens. Stale leases are automatically reclaimed — no stuck jobs. + - icon: 🧹 + title: Auto Cleanup + details: Set retainRuns to automatically purge old completed runs. Or call purgeRuns() for manual batch cleanup. --- diff --git a/website/public/llms.txt b/website/public/llms.txt index ba4b4886..553cf38e 100644 --- a/website/public/llms.txt +++ b/website/public/llms.txt @@ -4,7 +4,7 @@ ## Overview -Durably is a minimal workflow engine that persists step results to SQLite or PostgreSQL. If a job is interrupted (server restart, browser tab close, crash), it automatically resumes from the last successful step. Supports libSQL/Turso (single-server, serverless), PostgreSQL (multi-worker), and SQLocal (browser/OPFS). +Durably is a minimal workflow engine that persists step results to SQLite or PostgreSQL. If a job is interrupted (server restart, browser tab close, crash), it automatically resumes from the last successful step. Supports libSQL/Turso (single-server or serverless), PostgreSQL (recommended for multi-worker), and SQLocal (browser/OPFS). ## Installation