feat(runtime)!: launch workflow children as their own runs

.changeset/blob-offload.md

@@ -0,0 +1,5 @@
+---
+'@swirls/looms': minor
+---
+
+`createLooms({ blobStore })` and `LoomsDurableObject` `blobs` store strings over 64 KiB by SHA-256 and keep a reference in the event or snapshot. A payload that is still over the 1 MiB event limit is stored as one blob. Reads, subscribers, folds, projections, and snapshot loads resolve the reference, so a run can complete with a payload larger than that limit. Trim removes a blob body once nothing retained still points at it. `memoryBlobStore`, `filesystemBlobStore`, and `r2BlobStore` are the included stores. Without a blob store, events over 1 MiB and snapshots over 8 MiB are still rejected.

.changeset/child-runs.md

@@ -0,0 +1,9 @@
+---
+'@swirls/looms': minor
+---
+
+feat(runtime)!: launch workflow children as their own runs
+
+A workflow can set `childExecution: 'run'` so spawn, map, fanout, and loop start a child run instead of a thread. The child run id is stable across retries. When the child finishes it signals `runtime.child.settled`, which completes the parent's wait. `childClose` is `terminate` or `wait`. The default remains in-process threads.
+
+BREAKING CHANGE: `RunLauncher` implementations must provide `signal` and `cancel`.

.changeset/definition-resolver.md

@@ -0,0 +1,7 @@
+---
+'@swirls/looms': minor
+---
+
+Definitions can be loaded at start, spawn, and recovery through `definitionResolver`, with registered definitions checked first and an LRU cache in front. A missing definition fails the start before any event is written, and the accepted version stays pinned on the run. Evaluator definitions now include `version`, defaulting to `v1`, and lookup takes the name and version.
+
+BREAKING CHANGE: `EvaluatorDefinition` requires `version`, and evaluator lookup is `(name, version)` instead of `(name)`. `defineEvaluator` still defaults the version to `v1`.

.changeset/effect-reliability.md

@@ -0,0 +1,9 @@
+---
+'@swirls/looms': minor
+---
+
+Worker callbacks now answer `{ status: 'accepted' | 'stale' | 'cancelled', cancelRequested }` instead of the run state. `createLooms().workerCallback` and `createLoomsClient().workerCallback` return that result. `cancelRequested` tells an attempt that is already running to stop, because a dispatch that has left the host cannot be recalled. A stale report records nothing.
+
+`failEffect({ error, nonRetryable, retryAfterMs })` fails the current attempt. `nonRetryable` ends the effect even when the retry policy has attempts left. `retryAfterMs` waits at least that long before the next attempt and never shortens the policy backoff. Worker `fail` callbacks take the same two fields.
+
+A heartbeat may include `details`. The latest details are kept across retries and included on the next `EffectWorkerTask`, so a resumed attempt can continue from the recorded progress.

.changeset/run-context.md

@@ -0,0 +1,5 @@
+---
+'@swirls/looms': minor
+---
+
+Runs accept immutable `attributes` and a `secrets` context. Attributes are recorded on `runtime.run.started`, the run state, and the run summary, and are passed to `authorize` when present. Secrets stay in a side store: they are available to effect handlers and in-process worker dispatch, and they are not written to events, snapshots, or run responses. A worker whose transport persists tasks reads them with `POST /runs/:runId/effects/:effectId/:attempt/secrets` (`client.workerSecrets`), which only answers the attempt that is currently running.

.oxlintrc.json

@@ -85,7 +85,8 @@
       "files": [
         "apps/demo/src/server.ts",
         "packages/debugger/src/server/**/*.ts",
-        "packages/s2/src/lite.ts"
+        "packages/s2/src/lite.ts",
+        "packages/runtime/src/filesystem-blob-store.ts"
       ],
       "rules": {
         "effecttsgo/global-fetch": "off",

apps/docs/src/docs-manifest.json

@@ -830,7 +830,7 @@
         "level": 2
       }
     ],
-    "text": "Reliability and recovery Durability depends on the host and storage you choose. In-memory examples lose history on exit. SQLite persists locally; production actor hosts must preserve one writer per run and provide wake scheduling. Replay and recovery are different operations Historical replay folds recorded events to reconstruct state and does not dispatch effects. Recovery loads a snapshot and later events, then resumes unfinished work. An effect that lacks a recorded outcome may execute again. Looms does not promise exactly-once actions across arbitrary external services. A crash after an external action Looms requests an external action. The service performs it. The host stops before recording success. Recovery cannot infer success from the local log alone. Pass ctx.effectId as an idempotency key where the external API supports it. Check that the service retains the key for as long as you may need to retry the action. For unsupported APIs, store a durable operation ID, query the provider for the outcome, or reconcile before retrying. Never assume adding an arbitrary header makes an API idempotent. Retries and errors Effect retry policies count total attempts: maxAttempts: 3 means one initial attempt and up to two retries. Attempt starts and retry deadlines are recorded before the runtime waits, so a restart preserves the retry budget and backoff. Configure start, heartbeat, and execution timeouts for work delivered to external workers. Worker callbacks include the effect ID and attempt number; stale callbacks cannot complete a newer attempt. Exhaustion records runtime.effect.failed . Signals and webhook delivery Use the embed start/signal idempotencyKey option with a stable runId for retries of the same logical request. Verify webhook signatures and transform allowed webhook payloads into typed events on the server. Deduplicate by the provider delivery ID and correlate by run, thread, and domain identifiers. Retrying with a fresh key defeats deduplication. Interrupted model streams A model call can produce partial deltas before failing. Those deltas may already be recorded; resuming work may incur another model call and different output. Render partial content as provisional, inspect completion events, and decide how your application handles abandoned partial turns. The adapter does not guarantee exactly-once billing or identical regenerated text. Snapshots, retention, and projections A compatible snapshot bounds replay work; it is not a backup or a schema migration. Trimming events removes older audit and time-travel history and can prevent rebuilding new projections from the beginning. Define a retention and archive policy before enabling trimAfterSnapshot. Projectors run after local commit. Durable delivery stores a cursor per projector name and version, retries without advancing that cursor, and dead-letters exhausted batches. A run can finish before an external index or webhook catches up, so monitor lag and keep projector writes idempotent. Test recovery Test a restart while parked, an interruption during an external action, duplicate webhook delivery, and a deployment with existing runs. The testing guide provides the local restart test. Review versioning before changing definitions. ) }"
+    "text": "Reliability and recovery Durability depends on the host and storage you choose. In-memory examples lose history on exit. SQLite persists locally; production actor hosts must preserve one writer per run and provide wake scheduling. Replay and recovery are different operations Historical replay folds recorded events to reconstruct state and does not dispatch effects. Recovery loads a snapshot and later events, then resumes unfinished work. An effect that lacks a recorded outcome may execute again. Looms does not promise exactly-once actions across arbitrary external services. A crash after an external action Looms requests an external action. The service performs it. The host stops before recording success. Recovery cannot infer success from the local log alone. Pass ctx.effectId as an idempotency key where the external API supports it. Check that the service retains the key for as long as you may need to retry the action. For unsupported APIs, store a durable operation ID, query the provider for the outcome, or reconcile before retrying. Never assume adding an arbitrary header makes an API idempotent. Retries and errors Effect retry policies count total attempts: maxAttempts: 3 means one initial attempt and up to two retries. Attempt starts and retry deadlines are recorded before the runtime waits, so a restart preserves the retry budget and backoff. Configure start, heartbeat, and execution timeouts for work delivered to external workers. Worker callbacks include the effect ID and attempt number; stale callbacks cannot complete a newer attempt. Exhaustion records runtime.effect.failed . A handler can stop that budget early with failEffect and nonRetryable: true , or ask the next attempt to wait at least retryAfterMs without shortening the policy backoff. A worker heartbeat can carry details ; the next dispatch of that effect includes the latest details so the attempt can resume. The callback response is {'{ status, cancelRequested }'} , not the run state. cancelRequested tells an attempt that is already running to stop. Signals and webhook delivery Use the embed start/signal idempotencyKey option with a stable runId for retries of the same logical request. Verify webhook signatures and transform allowed webhook payloads into typed events on the server. Deduplicate by the provider delivery ID and correlate by run, thread, and domain identifiers. Retrying with a fresh key defeats deduplication. Interrupted model streams A model call can produce partial deltas before failing. Those deltas may already be recorded; resuming work may incur another model call and different output. Render partial content as provisional, inspect completion events, and decide how your application handles abandoned partial turns. The adapter does not guarantee exactly-once billing or identical regenerated text. Snapshots, retention, and projections A compatible snapshot bounds replay work; it is not a backup or a schema migration. Trimming events removes older audit and time-travel history and can prevent rebuilding new projections from the beginning. Define a retention and archive policy before enabling trimAfterSnapshot. Projectors run after local commit. Durable delivery stores a cursor per projector name and version, retries without advancing that cursor, and dead-letters exhausted batches. A run can finish before an external index or webhook catches up, so monitor lag and keep projector writes idempotent. Test recovery Test a restart while parked, an interruption during an external action, duplicate webhook delivery, and a deployment with existing runs. The testing guide provides the local restart test. Review versioning before changing definitions. ) }"
   },
   {
     "path": "/docs/schedules",

apps/docs/src/routes/docs.reliability.tsx

@@ -67,7 +67,13 @@ function Page() {
         a restart preserves the retry budget and backoff. Configure start, heartbeat, and execution
         timeouts for work delivered to external workers. Worker callbacks include the effect ID and
         attempt number; stale callbacks cannot complete a newer attempt. Exhaustion records{' '}
-        <code>runtime.effect.failed</code>.
+        <code>runtime.effect.failed</code>. A handler can stop that budget early with{' '}
+        <code>failEffect</code> and <code>nonRetryable: true</code>, or ask the next attempt to wait
+        at least <code>retryAfterMs</code> without shortening the policy backoff. A worker heartbeat
+        can carry <code>details</code>; the next dispatch of that effect includes the latest details
+        so the attempt can resume. The callback response is{' '}
+        <code>{'{ status, cancelRequested }'}</code>, not the run state.{' '}
+        <code>cancelRequested</code> tells an attempt that is already running to stop.
       </p>
       <h2 id="signals-and-webhook-delivery">
         Signals and webhook delivery

docs/modules.md

@@ -44,6 +44,8 @@ Each module owns its definitions. `workflow({ definitions: [checkout] })` instal
 
 At initialization, Looms rejects duplicate `(kind, name)` registrations and definitions whose kind is not implemented by their owning module. Starting or spawning an unregistered definition fails before any run events are written. An unsupported thread kind is rejected the same way; if such an event somehow appears in a log, fold marks the thread failed instead of inventing empty state. Pass all child definitions to their owning modules too.
 
+`defineWorkflow({ childExecution: 'run' })` launches each spawned, mapped, or looped child as its own run instead of a thread in the parent. The child run id is derived from the parent run and the child thread id, so a retry finds the same run. The child appends `runtime.child.settled` to the parent when it finishes, and that event completes the parent's wait. `childClose: 'terminate'` (the default) cancels those runs when the parent is cancelled. `childClose: 'wait'` cancels the children first and cancels the parent only after they have finished. A missing `RunLauncher` fails the launch. The default `childExecution` is `'thread'`, which keeps the child inside the parent run.
+
 `defineModule(options, setup)` returns a complete runtime module. The callback receives typed `effect`, `projection`, `thread`, `input`, and `emit` builders. Only returned members are installed; access them through `payments.effects.charge` and `payments.projections.ledger`. Event schemas can be inline or supplied as a reusable `defineEventCatalog`.
 
 Modules that implement a custom thread kind can return `definitions` alongside `threads`. The runtime gathers these for `start`, HTTP starts, and cross-module child spawning.

docs/production.md

@@ -13,19 +13,77 @@ coalesced. A deployment, eviction, or alarm retry may repeat a wake; event
 preconditions make that safe. Cross-object queries and transactions are not
 provided. Build global run lists with a projector.
 
+A workflow can launch children with `childExecution: 'run'`. Each child is its own run, addressed by a deterministic id, and reports `runtime.child.settled` back to the parent. Cancelling the parent cancels those children. `childClose: 'wait'` records `runtime.child.close.requested` and cancels the parent only after the children reach a terminal status. `childClose: 'terminate'` cancels them as the parent closes. Children stay threads in the parent run unless `childExecution` is set.
+
 All built-in stores reject an event over 1 MiB and a snapshot over 8 MiB,
 measured as serialized UTF-8, before storage is mutated. The stable error names
 are `EventPayloadTooLargeError` and `SnapshotPayloadTooLargeError`.
 
+Pass `blobStore` to `createLooms`, or `blobs` to `LoomsDurableObject` /
+`durableObjectEventStore`, to keep larger JSON. Strings over 64 KiB are replaced
+with `{ "$looms": "blob", "sha256", "bytes", "mediaType": "application/json" }`
+before those limits run. That object is reserved in stored JSON. If an event is
+still over 1 MiB, its payload is stored as one blob. The body is SHA-256
+content, so identical JSON is stored once. `read`, live subscribers, folds,
+projections, and snapshot loads resolve references, including references nested
+inside another blob. Objects and arrays stay inline so snapshot state still
+decodes; a snapshot that is over 8 MiB after large strings are removed is still
+rejected. Trim deletes a body when no retained event and no latest snapshot in
+any run still needs it. Without a blob store, oversized events and snapshots
+are still rejected.
+
+`memoryBlobStore` is the in-memory implementation. `filesystemBlobStore(directory)`
+is the Bun and Node implementation. `r2BlobStore(bucket)` writes an R2 bucket
+under `looms/blobs/`. Every run's Durable Object can share one bucket. Each run
+writes a marker per body it references, so trimming one run checks only that
+body's markers, never other runs' logs. If a body is deleted while another run
+is referencing it, that run's append fails and its retry uploads the body again.
+
+## Run attributes and secret context
+
+`attributes` on `start` are immutable labels (tenant, project, org). They are
+written on `runtime.run.started`, folded into run state, and copied onto the
+run summary. Authorization receives them on `route.attributes` when a run has
+any. They must be safe for every reader of the run.
+
+`secrets` are bearer material for the run. The memory and SQLite stores keep
+them in a side table, `looms_run_secrets`, which is not part of the event log
+or a snapshot. Effect handlers read them on `ctx.secrets`. An in-process worker
+receives them on the dispatch task. A transport that persists that task must
+omit `secrets`. Its worker reads them with
+`POST /runs/:runId/effects/:effectId/:attempt/secrets` (`client.workerSecrets`)
+after its `started` callback is accepted. The route uses `worker` access, so it
+returns 503 until the host configures worker auth. It answers
+`{ status: 'accepted', secrets }` only while that attempt is started and
+unsettled, and `{ status: 'stale' }` or `{ status: 'cancelled' }` otherwise.
+Responses are sent with `cache-control: no-store`. A handler must not copy
+secrets into an emitted event.
+
+A retry of the same start may omit `secrets` and keeps the stored context. Sending
+a different context conflicts, and the error does not include the secret values.
+Forks and effect-launched runs inherit the parent's attributes and secrets.
+A store without `runSecrets` rejects a start that supplies secrets.
+
 ## Worker effect protocol
 
-1. The runtime dispatches `{ runId, effectId, attempt, type, input }`.
+1. The runtime dispatches `{ runId, effectId, attempt, type, input, details? }`.
+   `details` is the progress carried by the previous attempt's last heartbeat.
 2. The worker calls `started` before the schedule-to-start deadline.
-3. Long work calls `heartbeat`; each accepted heartbeat extends the heartbeat
-   deadline without exceeding start-to-close.
-4. The worker calls exactly one of `complete` or `fail`.
+3. Long work calls `heartbeat`. Each accepted heartbeat extends the heartbeat
+   deadline without exceeding start-to-close. A heartbeat may include `details`;
+   the latest details are kept across retries and sent on the next dispatch.
+4. The worker calls exactly one of `complete` or `fail`. `fail` may set
+   `nonRetryable` to stop the retry policy, or `retryAfterMs` to wait at least
+   that long before the next attempt (the policy backoff is never shortened).
+   A local handler does the same by failing with `failEffect`.
 5. On cancellation, stop side effects and call `cancelled`.
 
+Each callback response is `{ status: 'accepted' | 'stale' | 'cancelled', cancelRequested }`.
+The host no longer returns the run state. `cancelRequested` is how a worker learns
+that an attempt already in progress should stop: a queued dispatch cannot be recalled.
+`stale` means the report was for another attempt, or the attempt had already settled,
+and the host recorded nothing.
+
 Callbacks include the same `effectId` and `attempt`. Stale callbacks are ignored.
 A dispatch transport failure is recorded as `ambiguous`: an operator must first
 verify the destination's idempotency record, then use `retryEffect` or
@@ -97,6 +155,21 @@ and schemas needed by active old-version runs. If a new projector is bad, stop
 delivery, deploy a corrected new version, and rebuild from archive; do not move a
 checkpoint backward after source events were trimmed.
 
+## Resolving definitions
+
+Pass `definitionResolver` to `createLooms`, `createRuntime`, an actor host, or
+`LoomsDurableObjectConfig` when a definition is not in the module's static list.
+`resolve(kind, name, version)` returns the full definition object, or `undefined`
+when that version does not exist. Registered module definitions are checked
+first. Results, including misses, stay in an LRU cache.
+
+Start, spawn, and launch resolve before they write `runtime.thread.started`.
+A miss fails the start with `UnknownDefinitionError` and writes no events. The
+accepted version is pinned on `runtime.run.started` and `runtime.thread.started`.
+Before a wake replays a log, those pinned identities are resolved again so fold
+can read them synchronously. An unknown workflow version fails that thread.
+Evaluator definitions include `version`, which defaults to `v1`.
+
 ## Production configuration
 
 Configure durable storage, actor routing, worker and operations tokens, bounded