Add durable store runtime

[djgrant]

Summary

Adds the durable store proposal and implementation for YieldStar.

• Adds SPEC.md describing the durable store API, selector-tracked onChange, default read policy, and runtime work.
• Adds Standard Schema-compatible store definitions and public exports.
• Adds workflow-side step.store(...) with durable get, select, update, and selector-tracked onChange.
• Adds StepStoreWait so workflows can suspend on store changes and resume via waiter wakeups.
• Adds memory and SQLite store clients with versioned snapshots, idempotent updates, path diffing, waiter registration, and wakeups.
• Adds test utility support for external sdk.store(...) updates.

Design Decisions

SQLite storage model

The SQLite runtime stores durable state in three tables:

• stores: keyed by (store_name, store_id), with version and serialized JSON state.
• store_update_results: keyed by (store_name, store_id, idempotency_key), storing the serialized StoreUpdateResult for idempotent external and workflow updates.
• store_waiters: keyed by (store_name, store_id, execution_id, step_key), storing the workflow/event identity, since_version, and tracked read_paths.

This keeps the state row, idempotency ledger, and wakeup subscriptions separate. The split makes replay behavior explicit: store data can advance independently, while a repeated idempotency key returns the original update result without applying the mutation again.

Store updates and derived write paths

updateStore reads the current state, clones it into a mutable draft, runs the updater, validates the resulting state with the store schema, increments the version, and writes the new JSON state back to stores.

Changed paths are derived by diffStorePaths(previousState, nextState) rather than supplied by the caller. The diff recursively compares the old and new JSON-shaped state and records leaf paths that changed, normalizing array indexes into numeric path segments. This avoids exposing public paths on update while still giving the runtime enough detail to wake relevant waiters.

onChange implementation

store.onChange(...) is implemented as a normal step-runner generator until the selector does not match. It yields a StepKey, checks the step cache, reads the current store snapshot, and runs the selector through trackStoreSelector.

If the selector returns a truthy/non-null value, the step yields a StepResult and completes like any other durable step. If it does not match, the step registers a waiter containing the original workflow event, the store key, the step key, the current store version, and the selector read paths. It then yields StepStoreWait.

The workflow wrapper records StepStoreWait with stepDone: false. That is the key detail: on replay, the wait step is treated as incomplete, so the selector is evaluated again instead of returning a cached wait response.

Proxy read tracking

trackStoreSelector wraps the cloned snapshot state in JavaScript Proxy objects. Every string property read records the path that was accessed, then recursively wraps child objects so nested reads are tracked too. Proxies are cached in a WeakMap so the same object is not wrapped repeatedly.

The proxy traps set and deleteProperty and throws if a selector mutates state. Selector functions are never serialized; only the paths read during the last non-matching evaluation are stored in store_waiters.

Path intersection for wakeups

When a store update commits, the store client loads waiters for that (store_name, store_id) and compares each waiter's read_paths with the update's derived write paths using prefix matching. A waiter wakes when either path is a prefix of the other.

That means a selector that reads messages can wake when an update writes messages[0], and a selector that reads profile.name can wake when an update replaces profile. The wakeup is intentionally coarse; the workflow replay still re-runs the selector before returning a value.

Runtime wakeup flow

When a waiter matches, the runtime deletes the waiter row and calls schedulerClient.requestWakeUp(waiter.event). The stored event is the original workflow event serialized into store_waiters, so the normal task queue path can resume the same execution without a special store-specific worker entrypoint.

The resumed workflow replays from the start, hits the same onChange step key, sees the prior wait step as incomplete, reads the latest committed store state, and runs the selector again. If the selector now matches, it persists a StepResult and the workflow continues. If it still does not match, it registers a fresh waiter at the newer store version.

Memory runtime mirrors SQLite semantics

The memory runtime implements the same contract with maps: committed store records, update-result records, and waiter records. It uses the same core helpers for cloning, validation, path diffing, selector tracking, and path intersection. This gives fast workflow-level tests while keeping behavior aligned with the SQLite durable runtime.

End-to-End Flow

1. A workflow calls yield* step.store(StoreDef, { id, initial }); the step creates or loads the store through StoreClient and reconstructs a WorkflowStore handle on replay.
2. The workflow calls yield* store.onChange("next-message", state => state.messages.find(...)).
3. The selector is run against a proxy-wrapped snapshot. If no message is ready, the runtime records that the selector read messages and persists a store waiter.
4. The workflow yields StepStoreWait, which is cached as an incomplete step and returns control to the worker.
5. An external caller or another workflow calls store.update(...); the runtime applies the update, validates the result, increments the store version, records any idempotent update result, and derives changed paths from the before/after state.
6. The store client checks waiters for the same store. If a waiter's read paths intersect the changed paths, it deletes the waiter and enqueues the original workflow event.
7. The workflow replays, reaches the same onChange step, re-reads current store state, re-runs the selector, and either completes with a StepResult or waits again.

Notes

1. Async updaters inside SQLite transactions — updateStore holds BEGIN IMMEDIATE across await params.updater(draft) . If anyone does real async work in an updater, it holds the DB lock.

Validation

• bun test
• bun run bundle