fix(driver-memory): persistence opt-in again + move the remaining suites to in-memory SQLite (#4065)

.changeset/cli-stale-dist-tests-and-project-root.md

@@ -0,0 +1,42 @@
+---
+"@objectstack/runtime": patch
+"@objectstack/cli": patch
+---
+
+fix(runtime,cli): `projectRoot` reaches the metadata repository; stop compiling tests into the CLI's dist (#4065)
+
+Two defects behind the last of #4065's stray `.objectstack/` directories — the
+one under `packages/cli/`. Neither is cosmetic.
+
+**1. `projectRoot` only got half the stack.** `createStandaloneStack`'s
+`projectRoot` is documented as scoping a boot's on-disk state to the project
+folder "so different examples / apps don't share a single database by accident",
+and it did redirect the default sqlite database. But it was never passed to
+`MetadataPlugin`, whose `FileSystemRepository` kept rooting at `process.cwd()`.
+So one "project root" meant two different directories: a boot pointed at project
+A wrote `A/.objectstack/data/` and `<cwd>/.objectstack/metadata/`. It now
+forwards `rootDir`, and `bootSchemaStack` accepts a `projectRoot` to pass down
+(defaulting to `process.cwd()`, which is right for every real `os migrate` — the
+CLI runs from the project directory). The two migrate integration suites, which
+build a fixture project in a tempdir, now scope their boots to it.
+
+**2. The CLI compiled its own tests into `dist/` — and vitest ran them.**
+`tsconfig.build.json` included all of `src` with no exclude, so every
+`src/**/*.test.ts` was emitted as `dist/**/*.test.js`. Two consequences:
+
+- `files: ["dist"]` **published** them.
+- This package has no vitest config, so `vitest run` collected the compiled
+  copies alongside the sources: **81 test files and 849 tests where the sources
+  hold 58 and 581**. Every `src/` test also ran as a stale `dist/` twin built
+  from whatever the source said at the last build.
+
+That is not just noise — it silently defeats edits. A fix to a source test
+appeared not to work, because the run was still executing the pre-fix compiled
+duplicate; that is exactly how the `.objectstack` residue survived a correct
+fix long enough to look like a different bug. It also means a source test could
+be edited to pass while its stale twin kept asserting the old behaviour, and
+neither would be obviously wrong. Test files are now excluded from the build.
+
+No other package is affected: the rest build with `tsup`, which emits only
+declared entry points. Verified by scanning every `packages/*/dist` for
+`*.test.js` — the CLI was the only hit.

.changeset/memory-driver-opt-in-persistence.md

@@ -0,0 +1,82 @@
+---
+"@objectstack/driver-memory": major
+"@objectstack/spec": major
+"@objectstack/plugin-dev": patch
+---
+
+fix(driver-memory,spec): persistence is opt-in again — `new InMemoryDriver()` is pure in-memory (#4065)
+
+`InMemoryDriverConfig.persistence` defaulted to `'auto'`, and in Node.js `'auto'`
+means **file**. So a bare `new InMemoryDriver()` — the shape every caller in this
+repo used — silently wrote `.objectstack/data/memory-driver.json` into the process
+CWD and reloaded it on the next boot. The default is now `false`.
+
+**This restores the accepted design rather than replacing it.** #815, the issue
+that introduced the persistence capability, specified it as opt-in in requirement
+\#1 — "默认情况下不启用持久化（纯内存，行为不变）" — and listed
+`new InMemoryDriver()` under "纯内存" in its own config examples. The `'auto'`
+default was a drift from that spec.
+
+What let the drift survive is worth naming, because it is not "there was no
+test". `MemoryConfigSchema` *did* pin the default, and asserted `'auto'`; the
+driver honoured `'auto'`; so spec and implementation agreed, and the pair looked
+verified. What nothing checked was whether the value they agreed on was the one
+#815 accepted. The driver's own `persistence.test.ts` could not have caught it
+either — every case there passes `persistence` explicitly, so the omitted-value
+path was untested on the implementation side. Both sides are now covered: three
+behavioural tests in `persistence.test.ts` (no CWD write, no cross-instance row
+carry-over, opt-in still persists) and the flipped schema assertion.
+
+**The symptom this fixes.** `packages/runtime/src/datasource-autoconnect.test.ts`
+seeds two rows with fixed ids and asserts the exact set. Run 1 passed and wrote
+the rows to disk; run 2 loaded them back, appended two more, and failed with four
+rows; run N had 2N. CI never saw it — every job is a fresh clone, so every CI run
+is run 1 — but `pnpm test` twice in one working tree could only ever go green
+once. The persisted file's `created_at` values, one pair per run, were the proof.
+
+(#4083 fixed that particular suite from the factory side, and its regression
+test is kept as-is. The blast radius was wider than one suite, though: **every**
+bare `new InMemoryDriver()` inherited the default, so any code path constructing
+one directly wrote to its working directory. Unit tests should not have write
+side effects on the CWD at all.)
+
+**Migrating.** Callers that want durability now ask for it:
+
+```ts
+new InMemoryDriver()                        // pure in-memory (new default)
+new InMemoryDriver({ persistence: 'file' })  // Node.js, durable across restarts
+new InMemoryDriver({ persistence: 'local' }) // browser, durable across reloads
+new InMemoryDriver({ persistence: 'auto' })  // previous default behaviour
+```
+
+The `'auto'` / `'file'` / `'local'` / custom-adapter paths are unchanged; only
+the value used when `persistence` is omitted moved.
+
+**Relationship to #4083.** That issue fixed the same hazard one consumer at a
+time, and landed first: `createDefaultDatasourceDriverFactory` now passes
+`persistence: false` for a declared `{ driver: 'memory' }` datasource and scopes
+an opted-in destination *per datasource*, and the dev sqlite step-down's
+last-resort rung passes `false` too. Both are kept exactly as #4083 wrote them.
+This change closes the half they deliberately left open — a directly-constructed
+`new InMemoryDriver()` — which is the path that still wrote into the working
+directory of whatever process happened to build one.
+
+The two are complementary, not redundant. #4083's per-datasource scoping is
+still the only thing that expands `'auto'`/`'file'`/`'local'` into a destination
+carrying the datasource name, so two pools that DO opt in never alias one file;
+its explicit `false` becomes belt-and-braces, which is the right posture for a
+path that must never persist.
+
+`DevPlugin`'s driver is now explicitly `persistence: false`, matching the cache,
+queue, job, i18n, storage and search stubs it ships beside — it was the one piece
+of that stack that quietly outlived the process.
+
+**One claim trimmed, no behaviour attached.** The class docstring called this a
+"production-ready implementation of the ObjectStack Driver Protocol". It stores
+no constraints at all — `create()` is a `table.push()` and `syncSchema()` only
+allocates an array — so there is no primary key, uniqueness, `NOT NULL`, foreign
+key or column typing, and `bulkCreate` lands duplicate ids where a SQL driver
+raises a violation (the second finding in #4065). The docstring now says so, and
+points test authors at in-memory SQLite. Per Prime Directive #10 the fix for
+`declared ≠ enforced` is to implement it, trim the claim, or file it; with this
+driver moving to maintenance-only the claim is what goes.

.changeset/scaffolds-drop-memory-driver.md

@@ -0,0 +1,28 @@
+---
+"@objectstack/cli": patch
+"create-objectstack": patch
+---
+
+chore(cli,create-objectstack): scaffolds no longer name a driver (#4065)
+
+`os init` and the `create-objectstack` blank template both listed
+`@objectstack/driver-memory` in the generated `dependencies`. It was the only
+driver named, which read as an endorsement — "this is the driver your app runs
+on" — when it is in fact the **last-resort rung** of the dev step-down (native
+`better-sqlite3` → WASM SQLite → mingo). A new project's first impression of the
+data layer should not be the engine that enforces no primary keys, no
+uniqueness, no `NOT NULL` and no column types.
+
+It was also redundant: `@objectstack/runtime` already depends on `driver-sql`,
+`driver-sqlite-wasm` and `driver-memory`, and every script in both scaffolds runs
+through the CLI, which carries all four. Removing the line changes nothing a
+generated project can do — `objectstack dev` still resolves SQLite by default,
+and `OS_DATABASE_URL` still selects Postgres / MySQL / MongoDB.
+
+Docs updated to match: the "packages you depend on" table in *Your first project*
+no longer lists a driver row (it now says where drivers come from), and the
+Memory Driver section of *Database Drivers* documents the opt-in persistence
+default, carries a migration callout for the old `'auto'` behaviour, and points
+test authors at in-memory SQLite. That section also claimed "Data is lost when
+the process exits", which was simply false while `'auto'` was the default — it
+wrote a file into the working directory.

.changeset/tests-off-memory-driver.md

@@ -0,0 +1,50 @@
+---
+"@objectstack/runtime": patch
+"@objectstack/client": patch
+"@objectstack/metadata": patch
+---
+
+test(runtime,client,metadata): back the remaining suites with in-memory SQLite instead of the mingo driver (#4065)
+
+Ten test files used `InMemoryDriver` as a convenience backing store — somewhere
+for rows to go while the suite proved something else (REST routing, datasource
+auto-connect, the batch `$ref` contract, metadata history). They now run on
+`SqliteWasmDriver` at `:memory:`, the same engine `@objectstack/verify`'s
+`bootStack` already gives the dogfood gate: pure JS (no native build, CI-safe on
+any runner) and real SQL semantics.
+
+The point is fidelity, not tidiness. Production runs SQL, and mingo differs from
+it in ways that let a suite pass while the behaviour it stands for is broken.
+Every failure this migration produced was a fixture defect the memory driver had
+been absorbing:
+
+- **Tables were never created.** `driver.create()` on the memory driver is a
+  bare `table.push()` onto an auto-vivified array, so an object registered
+  *after* `kernel.bootstrap()` — which misses the boot-time schema sync — looked
+  fine. On SQL the first write fails with `no such table`, which the REST error
+  mapper turns into a **404 `OBJECT_NOT_FOUND`**: a routing-shaped symptom for a
+  DDL-shaped cause. Four suites needed an explicit `syncObjectSchema`.
+- **A missing object declaration read as working.** `notifications.hono.integration`
+  writes `sys_notification`, which `MessagingServicePlugin` does not declare —
+  it is a platform object, and that lean kernel never booted `platform-objects`.
+  Auto-vivification hid the omission entirely. The suite now registers the real
+  `SysNotification` rather than a hand-copied stand-in, so there is still exactly
+  one schema for it (Prime Directive #12).
+- **`connect()` was optional.** The memory driver needs none; a SQL driver does.
+
+What deliberately did NOT move: `read-coercion-conformance` keeps its two-driver
+matrix (proving a stored value reads back as its declared type on *both* engines
+is the entire point of that gate), and the suites whose subject IS the memory
+driver or its wiring — `standalone-stack` (`memory://` scheme),
+`sqlite-driver-fallback` (the dev step-down), the CLI's driver-label tests, and
+driver-memory's own suite.
+
+`datasource-autoconnect` is in that second group as of #4083, which landed a
+regression test there for exactly the memory-pool property this PR originally
+proposed to migrate away from. Moving that file to SQLite would have left the
+new test passing vacuously — a wasm-SQLite pool never writes `.objectstack/` at
+all — so it stays on the memory driver and keeps guarding what it was written
+to guard.
+
+No new coverage is claimed here: each suite asserts exactly what it asserted
+before, against a more faithful store.

content/docs/data-modeling/drivers.mdx

@@ -353,34 +353,62 @@ equivalent age-based reap.
 ## Memory Driver
 
 The in-memory driver keeps records in plain in-process objects (queried via
-[`mingo`](https://github.com/kofrasa/mingo)).
+[`mingo`](https://github.com/kofrasa/mingo)). Data is lost when the process
+exits unless persistence is explicitly requested.
 It is the **last-resort fallback** in dev mode: `objectstack dev` prefers native
 SQLite (`better-sqlite3`), falls back to the pure-JS WASM SQLite driver if the
 native binary is unavailable, and only drops to the in-memory driver if WASM also
 fails to load. Set `OS_DATABASE_DRIVER=memory` to select it explicitly.
 
-**Whether data survives the process depends on how the driver is built** (#4083):
+**Both ways of building the driver are ephemeral by default** (#4083, #4065):
 
 | How it is built | Persistence |
 | :--- | :--- |
 | A **declared datasource** — `{ driver: 'memory' }` in a stack/app config | **Ephemeral.** Nothing is written to disk unless the declaration sets `config.persistence` — and when it does, the destination is scoped per datasource, so two memory datasources never share one file. |
-| `new InMemoryDriver()` **constructed directly** | `persistence: 'auto'` — under Node that means a JSON file at `.objectstack/data/memory-driver.json`, relative to the process's working directory, reloaded on the next boot. |
-
-Pass `persistence: false` for a driver you construct yourself and want purely in
-memory — a test, for instance, which should neither depend on nor leave behind
-state in the working directory. Two directly-constructed `'auto'` drivers in one
-process still share that single default path.
+| `new InMemoryDriver()` **constructed directly** | **Ephemeral.** Pass `persistence` to opt in (see below). Note the scoping above is the *factory's* job: two directly-constructed `'auto'` drivers in one process still share the single default path. |
 
 ```typescript
 import { InMemoryDriver } from '@objectstack/driver-memory';
 
-new InMemoryDriver({ persistence: false });   // pure memory
-new InMemoryDriver();                         // 'auto' — file-backed under Node
+new InMemoryDriver();                         // pure memory — the default
+new InMemoryDriver({ persistence: 'file' });  // opt in to durability
+```
+
+### Persistence is opt-in
+
+A bare `new InMemoryDriver()` persists nothing. Durability is requested
+explicitly:
+
+```typescript
+new InMemoryDriver({ persistence: 'file' })   // Node.js — .objectstack/data/memory-driver.json
+new InMemoryDriver({ persistence: 'local' })  // browser — localStorage
+new InMemoryDriver({ persistence: 'auto' })   // pick per environment (file / localStorage / off)
 ```
 
+`'auto'` chooses localStorage in a browser, a file under Node.js, and **disables**
+persistence in serverless/edge runtimes (Vercel, Lambda, Netlify, Cloud Run, Deno
+Deploy) where a local write would be silently discarded — supply a custom adapter
+there instead.
+
+<Callout type="warn">
+Before v17 the default was `'auto'`, which on Node.js meant **file** — so a bare
+`new InMemoryDriver()` silently wrote `.objectstack/data/memory-driver.json` into
+the working directory and reloaded it on the next boot. If you relied on that,
+pass `persistence: 'auto'` (or `'file'`) explicitly. See
+[#4065](https://github.com/objectstack-ai/objectstack/issues/4065).
+</Callout>
+
 <Callout type="tip">
-Use the memory driver for unit tests. It requires no setup and runs instantly —
-with `persistence: false`, so one run cannot see what an earlier run left behind.
+For tests, prefer in-memory **SQLite** — `SqlDriver` with
+`connection: { filename: ':memory:' }`, or `SqliteWasmDriver({ filename: ':memory:' })`
+when you want no native build. Both give the SQL semantics production runs on;
+mingo does not enforce primary keys, uniqueness, `NOT NULL` or column types, so a
+green run against the memory driver is weaker evidence than it looks. The
+framework's own dogfood gate boots on WASM SQLite at `:memory:` for this reason.
+
+The memory driver remains fine where you want no setup at all and the assertions
+do not depend on storage semantics — it is ephemeral by default, so one run
+cannot see what an earlier run left behind.
 </Callout>
 
 ## Local Environment Runtime

content/docs/getting-started/your-first-project.mdx

@@ -164,12 +164,16 @@ framework surface you install:
 | Package | What it does |
 |:---|:---|
 | `@objectstack/spec` | The protocol: `defineStack`, `ObjectSchema`, `Field.*` builders, all Zod schemas. Your metadata imports only this. |
-| `@objectstack/runtime` | The kernel that interprets metadata at runtime. |
-| `@objectstack/driver-memory` | In-memory database driver for development. Swap for SQLite / Postgres / MongoDB in production — [no code changes](/docs/data-modeling/drivers). |
+| `@objectstack/runtime` | The kernel that interprets metadata at runtime. Brings the database drivers with it — you do not install one separately. |
 | `@objectstack/plugin-hono-server` | The HTTP server that mounts the generated REST API. |
 | `@objectstack/connector-rest` · `-openapi` · `-mcp` | The three generic connector executors — register the `rest` / `openapi` / `mcp` provider factories so declarative `connectors:` entries materialize ([ADR-0097](/docs/automation/flows)). |
 | `@objectstack/cli` *(dev)* | The `os` / `objectstack` CLI: `dev`, `validate`, `build`, `start`. |
 
+No driver package is listed because none is installed directly: `@objectstack/runtime`
+depends on `driver-sql`, `driver-sqlite-wasm` and `driver-memory`, and the CLI carries
+them too. `objectstack dev` resolves **SQLite** by default; point `OS_DATABASE_URL` at
+Postgres / MySQL / MongoDB for production — [no code changes](/docs/data-modeling/drivers).
+
 ## 3. Run it
 
 ```bash

packages/cli/src/commands/init.ts

@@ -103,11 +103,17 @@ export const TEMPLATES: Record<string, {
     description: 'Full application with objects, views, and actions',
     get dependencies() {
       const v = pkgVersion();
+      // No driver is listed on purpose. `@objectstack/runtime` already depends
+      // on driver-sql / driver-sqlite-wasm / driver-memory, and every script
+      // here runs through the CLI, which carries them too — so naming one was
+      // redundant. Naming `driver-memory` specifically also read as an
+      // endorsement: it is the LAST-RESORT rung of the dev step-down (native
+      // better-sqlite3 → wasm SQLite → mingo), not the driver a new app should
+      // start on. `objectstack dev` resolves sqlite by default.
       return {
         '@objectstack/spec': v,
         '@objectstack/runtime': v,
         '@objectstack/objectql': v,
-        '@objectstack/driver-memory': v,
       };
     },
     get devDependencies() {